[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575267#comment-16575267 ] Thomas Graves commented on SPARK-25024: --- ok, I'm not familiar with mesos hardly at all so I

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574956#comment-16574956 ] Thomas Graves commented on SPARK-23207: --- ok, I guess I disagree with that. Any correctness bug is

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574886#comment-16574886 ] Thomas Graves commented on SPARK-23207: --- [~jiangxb1987] ^ > Shuffle+Repartition on an DataFrame

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573221#comment-16573221 ] Thomas Graves commented on SPARK-24924: --- | There was a discussion about why we shouldn't support

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572232#comment-16572232 ] Thomas Graves commented on SPARK-23207: --- does this affect spark 2.2 and < ? from the description

[jira] [Commented] (SPARK-24598) SPARK SQL:Datatype overflow conditions gives incorrect result

[ https://issues.apache.org/jira/browse/SPARK-24598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572011#comment-16572011 ] Thomas Graves commented on SPARK-24598: --- In the very least we should file a separate Jira to track

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571908#comment-16571908 ] Thomas Graves commented on SPARK-24924: --- so originally when I started on this I didn't know about

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571852#comment-16571852 ] Thomas Graves commented on SPARK-24924: --- thanks, I missed it in the output for spark as I was just

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570840#comment-16570840 ] Thomas Graves commented on SPARK-24924: --- so officially the spark api compatibility is only at the

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570736#comment-16570736 ] Thomas Graves commented on SPARK-24924: --- so if the user includes the databricks jar and they

[jira] [Resolved] (SPARK-24992) spark should randomize yarn local dir selection

[ https://issues.apache.org/jira/browse/SPARK-24992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24992. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > spark

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570638#comment-16570638 ] Thomas Graves commented on SPARK-24924: --- So something I just thought of that I want to clarify, is

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570586#comment-16570586 ] Thomas Graves commented on SPARK-24924: --- For compatibility we can't remove it unless major

[jira] [Resolved] (SPARK-24981) ShutdownHook timeout causes job to fail when succeeded when SparkContext stop() not called by user program

[ https://issues.apache.org/jira/browse/SPARK-24981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24981. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 >

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570220#comment-16570220 ] Thomas Graves commented on SPARK-24924: --- {quote}I have followed the changes in Avro and I don't

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570194#comment-16570194 ] Thomas Graves commented on SPARK-25024: --- We need to make it clear what mesos supports for security

[jira] [Commented] (SPARK-25023) Clarify Spark security documentation

[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568805#comment-16568805 ] Thomas Graves commented on SPARK-25023: --- note some of this was already updated with

[jira] [Issue Comment Deleted] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25024: -- Comment: was deleted (was: I'm going to work on this.) > Update mesos documentation to be

[jira] [Commented] (SPARK-25023) Clarify Spark security documentation

[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568767#comment-16568767 ] Thomas Graves commented on SPARK-25023: --- I'm going to work on this > Clarify Spark security

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568766#comment-16568766 ] Thomas Graves commented on SPARK-25024: --- I'm going to work on this. > Update mesos documentation

[jira] [Created] (SPARK-25024) Update mesos documentation to be clear about security supported

Thomas Graves created SPARK-25024: - Summary: Update mesos documentation to be clear about security supported Key: SPARK-25024 URL: https://issues.apache.org/jira/browse/SPARK-25024 Project: Spark

[jira] [Created] (SPARK-25023) Clarify Spark security documentation

Thomas Graves created SPARK-25023: - Summary: Clarify Spark security documentation Key: SPARK-25023 URL: https://issues.apache.org/jira/browse/SPARK-25023 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-25016) remove Support for hadoop 2.6

Thomas Graves created SPARK-25016: - Summary: remove Support for hadoop 2.6 Key: SPARK-25016 URL: https://issues.apache.org/jira/browse/SPARK-25016 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-25016) remove Support for hadoop 2.6

[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25016: -- Target Version/s: 3.0.0 > remove Support for hadoop 2.6 > - > >

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568393#comment-16568393 ] Thomas Graves commented on SPARK-24924: --- | It wouldn't be very different for 2.4.0. It could be

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568204#comment-16568204 ] Thomas Graves commented on SPARK-24924: --- [~felixcheung] did your discussion on the same thing with

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568199#comment-16568199 ] Thomas Graves commented on SPARK-24924: --- Hmm, so we are adding this for ease of upgrading I guess

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Target Version/s: 2.4.0 > Spark scheduler can hang when fetch failures, executor lost, task

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567045#comment-16567045 ] Thomas Graves commented on SPARK-24924: --- why are we doing this? If a user ships the spark-avro

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565513#comment-16565513 ] Thomas Graves commented on SPARK-24909: --- looking more I think the fix may actually just be to

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565437#comment-16565437 ] Thomas Graves commented on SPARK-24909: --- this is unfortunately not a straight forward fix, the

[jira] [Commented] (SPARK-24986) OOM in BufferHolder during writes to a stream

[ https://issues.apache.org/jira/browse/SPARK-24986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565296#comment-16565296 ] Thomas Graves commented on SPARK-24986: --- fyi [~irashid] I know you were looking at memory related

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563711#comment-16563711 ] Thomas Graves commented on SPARK-24615: --- so I guess my question is this the right approach at all. 

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563689#comment-16563689 ] Thomas Graves commented on SPARK-24579: --- going from Spark feeds data into DL/AI frameworks for

[jira] [Commented] (SPARK-24934) Complex type and binary type in in-memory partition pruning does not work due to missing upper/lower bounds cases

[ https://issues.apache.org/jira/browse/SPARK-24934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561944#comment-16561944 ] Thomas Graves commented on SPARK-24934: --- what is the real affected versions here?  Since it went

[jira] [Updated] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23243: -- Priority: Blocker (was: Major) > Shuffle+Repartition on an RDD could lead to incorrect

[jira] [Resolved] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13343. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 >

[jira] [Commented] (SPARK-24918) Executor Plugin API

[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558331#comment-16558331 ] Thomas Graves commented on SPARK-24918: --- I think this is a good idea. I thought I had seen a Jira

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554803#comment-16554803 ] Thomas Graves commented on SPARK-24909: --- I haven't come up with a fix yet but have been looking at

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Summary: Spark scheduler can hang when fetch failures, executor lost, task running on lost

[jira] [Commented] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554701#comment-16554701 ] Thomas Graves commented on SPARK-24909: --- Note this may have been introduced as part of SPARK-23433

[jira] [Updated] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and

[jira] [Created] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

Thomas Graves created SPARK-24909: - Summary: Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts Key: SPARK-24909 URL: https://issues.apache.org/jira/browse/SPARK-24909

[jira] [Updated] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Priority: Critical (was: Major) > Spark scheduler can hang with fetch failures and executor

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554327#comment-16554327 ] Thomas Graves commented on SPARK-24615: --- Right so I think part of this is trying to make it more

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554306#comment-16554306 ] Thomas Graves commented on SPARK-23128: --- we also did some initial evaluation with it as well and

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551299#comment-16551299 ] Thomas Graves commented on SPARK-23128: --- [~carsonwang]  I'm curious if you are still running with

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551240#comment-16551240 ] Thomas Graves commented on SPARK-24615: --- the other thing which I think I mentioned above is could

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551233#comment-16551233 ] Thomas Graves commented on SPARK-24615: --- Ok so thinking about this a bit more I slightly misread

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550803#comment-16550803 ] Thomas Graves commented on SPARK-24615: --- yes if any requirement can't be satisfied it would use

[jira] [Assigned] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24755: - Assignee: Hieu Tri Huynh > Executor loss can cause task to not be resubmitted >

[jira] [Resolved] (SPARK-24755) Executor loss can cause task to not be resubmitted

[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24755. --- Resolution: Fixed Fix Version/s: 2.3.3 2.4.0 > Executor loss can

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549356#comment-16549356 ] Thomas Graves commented on SPARK-22151: --- ok thanks, must have missed that. > PYTHONPATH not

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549348#comment-16549348 ] Thomas Graves commented on SPARK-22151: --- [~srowen] why did you blank out fixed version and

[jira] [Resolved] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-22151. --- Resolution: Fixed Fix Version/s: 2.4.0 > PYTHONPATH not picked up from the

[jira] [Comment Edited] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549345#comment-16549345 ] Thomas Graves edited comment on SPARK-24615 at 7/19/18 2:28 PM: but my

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549345#comment-16549345 ] Thomas Graves commented on SPARK-24615: --- but my point is exactly that, it shouldn't be yet another

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549266#comment-16549266 ] Thomas Graves commented on SPARK-24615: --- I think the usage for cpu/memory is the same.  You know

[jira] [Assigned] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-22151: - Assignee: Parth Gandhi > PYTHONPATH not picked up from the spark.yarn.appMasterEnv

[jira] [Updated] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22151: -- Fix Version/s: 2.4.0 > PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly >

[jira] [Updated] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24677: -- Fix Version/s: 2.2.3 > TaskSetManager not updating successfulTaskDurations for old stage

[jira] [Resolved] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24677. --- Resolution: Fixed Fix Version/s: 2.4.0 2.3.3 > TaskSetManager not

[jira] [Assigned] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24677: - Assignee: dzcxzl > TaskSetManager not updating successfulTaskDurations for old stage

[jira] [Updated] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24677: -- Summary: TaskSetManager not updating successfulTaskDurations for old stage attempts (was:

[jira] [Comment Edited] (SPARK-24677) Avoid NoSuchElementException from MedianHeap

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548210#comment-16548210 ] Thomas Graves edited comment on SPARK-24677 at 7/18/18 6:22 PM: This is

[jira] [Commented] (SPARK-24677) Avoid NoSuchElementException from MedianHeap

[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548210#comment-16548210 ] Thomas Graves commented on SPARK-24677: --- In this case one of the older stage attempts (that is a

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547903#comment-16547903 ] Thomas Graves commented on SPARK-24615: --- did the design doc permissions change? I can't seem to

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546617#comment-16546617 ] Thomas Graves commented on SPARK-24615: --- ok, I agree, I think this SPIP should at least cover how

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546580#comment-16546580 ] Thomas Graves commented on SPARK-24615: --- The user is responsible for asking yarn for the right

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545253#comment-16545253 ] Thomas Graves commented on SPARK-24615: --- [~jerryshao] ^ > Accelerator aware task scheduling for

[jira] [Resolved] (SPARK-24610) wholeTextFiles broken for small files

[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24610. --- Resolution: Fixed Fix Version/s: 2.4.0 > wholeTextFiles broken for small files >

[jira] [Assigned] (SPARK-24610) wholeTextFiles broken for small files

[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24610: - Assignee: Dhruve Ashar > wholeTextFiles broken for small files >

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537035#comment-16537035 ] Thomas Graves commented on SPARK-16534: --- I agree it seems a bit of a bad user story to drop

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537034#comment-16537034 ] Thomas Graves commented on SPARK-16534: --- If we aren't going to do this we should close this as

[jira] [Updated] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13343: -- Description: Currently Speculative tasks that didn't commit can show up as success 

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529869#comment-16529869 ] Thomas Graves commented on SPARK-17181: --- that would be a question for [~marymwu] > [Spark2.0 web

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528293#comment-16528293 ] Thomas Graves commented on SPARK-24615: --- maybe I"m missing it but how is this working with the

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527865#comment-16527865 ] Thomas Graves commented on SPARK-23309: --- We tried this on a newest 2.3.1 and haven't been able to

[jira] [Resolved] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-23309. --- Resolution: Works for Me > Spark 2.3 cached query performance 20-30% worse then spark 2.2 >

[jira] [Resolved] (SPARK-24372) Create script for preparing RCs

[ https://issues.apache.org/jira/browse/SPARK-24372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24372. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0 > Create

[jira] [Resolved] (SPARK-24519) MapStatus has 2000 hardcoded

[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24519. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > MapStatus

[jira] [Updated] (SPARK-22897) Expose stageAttemptId in TaskContext

[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22897: -- Fix Version/s: 2.2.2 > Expose stageAttemptId in TaskContext >

[jira] [Resolved] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24589. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:02 PM: more

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:01 PM: more

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves commented on SPARK-24552: --- more details on hadoop committer side: So I think the

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519430#comment-16519430 ] Thomas Graves commented on SPARK-24552: --- this is actually a problem with hadoop committers, v1 and

[jira] [Commented] (SPARK-24611) Clean up OutputCommitCoordinator

[ https://issues.apache.org/jira/browse/SPARK-24611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519420#comment-16519420 ] Thomas Graves commented on SPARK-24611: --- [~joshrosen]  just noticed you were the last one to

[jira] [Commented] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

[ https://issues.apache.org/jira/browse/SPARK-24622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519416#comment-16519416 ] Thomas Graves commented on SPARK-24622: --- Need to investigate further/test to make sure I am not

[jira] [Created] (SPARK-24622) Task attempts in other stage attempts not killed when one task attempt succeeds

Thomas Graves created SPARK-24622: - Summary: Task attempts in other stage attempts not killed when one task attempt succeeds Key: SPARK-24622 URL: https://issues.apache.org/jira/browse/SPARK-24622

[jira] [Updated] (SPARK-24519) MapStatus has 2000 hardcoded

[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24519: -- Description: MapStatus uses hardcoded value of 2000 partitions to determine if it should use

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Affects Version/s: 2.2.0 2.2.1 2.3.0

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Critical) > Task attempt numbers are reused when stages are retried

[jira] [Commented] (SPARK-22148) TaskSetManager.abortIfCompletelyBlacklisted should not abort when all current executors are blacklisted but dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-22148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512691#comment-16512691 ] Thomas Graves commented on SPARK-22148: --- ok, just update if you start working on it. thanks. >

[jira] [Commented] (SPARK-24539) HistoryServer does not display metrics from tasks that complete after stage failure

[ https://issues.apache.org/jira/browse/SPARK-24539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512643#comment-16512643 ] Thomas Graves commented on SPARK-24539: --- Its possible, I thought when I checked the history server

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512519#comment-16512519 ] Thomas Graves commented on SPARK-24552: --- sorry just realized the v2 api is still marked experiment

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Critical (was: Blocker) > Task attempt numbers are reused when stages are retried

[jira] [Commented] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16512504#comment-16512504 ] Thomas Graves commented on SPARK-24552: --- Note if this is a correctness bug and can cause data

[jira] [Updated] (SPARK-24552) Task attempt numbers are reused when stages are retried

[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24552: -- Priority: Blocker (was: Major) > Task attempt numbers are reused when stages are retried >

<    2   3   4   5   6   7   8   9   10   11   >