[jira] [Updated] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple time

2018-08-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25250: -- Description: We recently had a scenario where a race condition occurred when a task from

[jira] [Updated] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple time

2018-08-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25250: -- Priority: Major (was: Minor) > Race condition with tasks running when new attempt for same

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-08-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-08-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581608#comment-16581608 ] Thomas Graves commented on SPARK-24924: --- I'd be ok with that but CSV has been that way already for

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581117#comment-16581117 ] Thomas Graves commented on SPARK-24924: --- [~cloud_fan] [~hyukjin.kwon] seems no one else has a

[jira] [Assigned] (SPARK-25043) spark-sql should print the appId and master on startup

2018-08-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-25043: - Assignee: Alessandro Bellina > spark-sql should print the appId and master on startup

[jira] [Resolved] (SPARK-25043) spark-sql should print the appId and master on startup

2018-08-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-25043. --- Resolution: Fixed Fix Version/s: 2.4.0 > spark-sql should print the appId and master

[jira] [Commented] (SPARK-24787) Events being dropped at an alarming rate due to hsync being slow for eventLogging

2018-08-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579854#comment-16579854 ] Thomas Graves commented on SPARK-24787: --- Yes it was caused by hsync, hsync has to go to the

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-08-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579851#comment-16579851 ] Thomas Graves commented on SPARK-24918: --- Personally I like the explicit config on better

[jira] [Updated] (SPARK-25051) where clause on dataset gives AnalysisException

2018-08-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25051: -- Priority: Blocker (was: Major) > where clause on dataset gives AnalysisException >

[jira] [Commented] (SPARK-23298) distinct.count on Dataset/DataFrame yields non-deterministic results

2018-08-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576709#comment-16576709 ] Thomas Graves commented on SPARK-23298: --- [~mjukiewicz] have you tried spark with fix for

[jira] [Commented] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576700#comment-16576700 ] Thomas Graves commented on SPARK-25081: --- thanks, wanted to clarify since the description only

[jira] [Commented] (SPARK-25081) Nested spill in ShuffleExternalSorter may access a released memory page

2018-08-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16576234#comment-16576234 ] Thomas Graves commented on SPARK-25081: --- Does this ever result in the task reading the wrong data

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16575267#comment-16575267 ] Thomas Graves commented on SPARK-25024: --- ok, I'm not familiar with mesos hardly at all so I

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574956#comment-16574956 ] Thomas Graves commented on SPARK-23207: --- ok, I guess I disagree with that. Any correctness bug is

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574886#comment-16574886 ] Thomas Graves commented on SPARK-23207: --- [~jiangxb1987] ^ > Shuffle+Repartition on an DataFrame

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16573221#comment-16573221 ] Thomas Graves commented on SPARK-24924: --- | There was a discussion about why we shouldn't support

[jira] [Commented] (SPARK-23207) Shuffle+Repartition on an DataFrame could lead to incorrect answers

2018-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572232#comment-16572232 ] Thomas Graves commented on SPARK-23207: --- does this affect spark 2.2 and < ? from the description

[jira] [Commented] (SPARK-24598) SPARK SQL:Datatype overflow conditions gives incorrect result

2018-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16572011#comment-16572011 ] Thomas Graves commented on SPARK-24598: --- In the very least we should file a separate Jira to track

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571908#comment-16571908 ] Thomas Graves commented on SPARK-24924: --- so originally when I started on this I didn't know about

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16571852#comment-16571852 ] Thomas Graves commented on SPARK-24924: --- thanks, I missed it in the output for spark as I was just

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570840#comment-16570840 ] Thomas Graves commented on SPARK-24924: --- so officially the spark api compatibility is only at the

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570736#comment-16570736 ] Thomas Graves commented on SPARK-24924: --- so if the user includes the databricks jar and they

[jira] [Resolved] (SPARK-24992) spark should randomize yarn local dir selection

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24992. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > spark

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570638#comment-16570638 ] Thomas Graves commented on SPARK-24924: --- So something I just thought of that I want to clarify, is

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570586#comment-16570586 ] Thomas Graves commented on SPARK-24924: --- For compatibility we can't remove it unless major

[jira] [Resolved] (SPARK-24981) ShutdownHook timeout causes job to fail when succeeded when SparkContext stop() not called by user program

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24981. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 >

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570220#comment-16570220 ] Thomas Graves commented on SPARK-24924: --- {quote}I have followed the changes in Avro and I don't

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16570194#comment-16570194 ] Thomas Graves commented on SPARK-25024: --- We need to make it clear what mesos supports for security

[jira] [Commented] (SPARK-25023) Clarify Spark security documentation

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568805#comment-16568805 ] Thomas Graves commented on SPARK-25023: --- note some of this was already updated with

[jira] [Issue Comment Deleted] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25024: -- Comment: was deleted (was: I'm going to work on this.) > Update mesos documentation to be

[jira] [Commented] (SPARK-25023) Clarify Spark security documentation

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568767#comment-16568767 ] Thomas Graves commented on SPARK-25023: --- I'm going to work on this > Clarify Spark security

[jira] [Commented] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568766#comment-16568766 ] Thomas Graves commented on SPARK-25024: --- I'm going to work on this. > Update mesos documentation

[jira] [Created] (SPARK-25024) Update mesos documentation to be clear about security supported

2018-08-03 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-25024: - Summary: Update mesos documentation to be clear about security supported Key: SPARK-25024 URL: https://issues.apache.org/jira/browse/SPARK-25024 Project: Spark

[jira] [Created] (SPARK-25023) Clarify Spark security documentation

2018-08-03 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-25023: - Summary: Clarify Spark security documentation Key: SPARK-25023 URL: https://issues.apache.org/jira/browse/SPARK-25023 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-25016) remove Support for hadoop 2.6

2018-08-03 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-25016: - Summary: remove Support for hadoop 2.6 Key: SPARK-25016 URL: https://issues.apache.org/jira/browse/SPARK-25016 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-25016) remove Support for hadoop 2.6

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-25016: -- Target Version/s: 3.0.0 > remove Support for hadoop 2.6 > - > >

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568393#comment-16568393 ] Thomas Graves commented on SPARK-24924: --- | It wouldn't be very different for 2.4.0. It could be

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568204#comment-16568204 ] Thomas Graves commented on SPARK-24924: --- [~felixcheung] did your discussion on the same thing with

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568199#comment-16568199 ] Thomas Graves commented on SPARK-24924: --- Hmm, so we are adding this for ease of upgrading I guess

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-08-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Target Version/s: 2.4.0 > Spark scheduler can hang when fetch failures, executor lost, task

[jira] [Commented] (SPARK-24924) Add mapping for built-in Avro data source

2018-08-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567045#comment-16567045 ] Thomas Graves commented on SPARK-24924: --- why are we doing this? If a user ships the spark-avro

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-08-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565513#comment-16565513 ] Thomas Graves commented on SPARK-24909: --- looking more I think the fix may actually just be to

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-08-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565437#comment-16565437 ] Thomas Graves commented on SPARK-24909: --- this is unfortunately not a straight forward fix, the

[jira] [Commented] (SPARK-24986) OOM in BufferHolder during writes to a stream

2018-08-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565296#comment-16565296 ] Thomas Graves commented on SPARK-24986: --- fyi [~irashid] I know you were looking at memory related

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563711#comment-16563711 ] Thomas Graves commented on SPARK-24615: --- so I guess my question is this the right approach at all. 

[jira] [Commented] (SPARK-24579) SPIP: Standardize Optimized Data Exchange between Spark and DL/AI frameworks

2018-07-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563689#comment-16563689 ] Thomas Graves commented on SPARK-24579: --- going from Spark feeds data into DL/AI frameworks for

[jira] [Commented] (SPARK-24934) Complex type and binary type in in-memory partition pruning does not work due to missing upper/lower bounds cases

2018-07-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16561944#comment-16561944 ] Thomas Graves commented on SPARK-24934: --- what is the real affected versions here?  Since it went

[jira] [Updated] (SPARK-23243) Shuffle+Repartition on an RDD could lead to incorrect answers

2018-07-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-23243: -- Priority: Blocker (was: Major) > Shuffle+Repartition on an RDD could lead to incorrect

[jira] [Resolved] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

2018-07-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13343. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 >

[jira] [Commented] (SPARK-24918) Executor Plugin API

2018-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558331#comment-16558331 ] Thomas Graves commented on SPARK-24918: --- I think this is a good idea. I thought I had seen a Jira

[jira] [Commented] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554803#comment-16554803 ] Thomas Graves commented on SPARK-24909: --- I haven't come up with a fix yet but have been looking at

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and

[jira] [Updated] (SPARK-24909) Spark scheduler can hang when fetch failures, executor lost, task running on lost executor, and multiple stage attempts

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Summary: Spark scheduler can hang when fetch failures, executor lost, task running on lost

[jira] [Commented] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554701#comment-16554701 ] Thomas Graves commented on SPARK-24909: --- Note this may have been introduced as part of SPARK-23433

[jira] [Updated] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Description: The DAGScheduler can hang if the executor was lost (due to fetch failure) and

[jira] [Created] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

2018-07-24 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-24909: - Summary: Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts Key: SPARK-24909 URL: https://issues.apache.org/jira/browse/SPARK-24909

[jira] [Updated] (SPARK-24909) Spark scheduler can hang with fetch failures and executor lost and multiple stage attempts

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24909: -- Priority: Critical (was: Major) > Spark scheduler can hang with fetch failures and executor

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554327#comment-16554327 ] Thomas Graves commented on SPARK-24615: --- Right so I think part of this is trying to make it more

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

2018-07-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554306#comment-16554306 ] Thomas Graves commented on SPARK-23128: --- we also did some initial evaluation with it as well and

[jira] [Commented] (SPARK-23128) A new approach to do adaptive execution in Spark SQL

2018-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551299#comment-16551299 ] Thomas Graves commented on SPARK-23128: --- [~carsonwang]  I'm curious if you are still running with

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551240#comment-16551240 ] Thomas Graves commented on SPARK-24615: --- the other thing which I think I mentioned above is could

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551233#comment-16551233 ] Thomas Graves commented on SPARK-24615: --- Ok so thinking about this a bit more I slightly misread

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550803#comment-16550803 ] Thomas Graves commented on SPARK-24615: --- yes if any requirement can't be satisfied it would use

[jira] [Assigned] (SPARK-24755) Executor loss can cause task to not be resubmitted

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24755: - Assignee: Hieu Tri Huynh > Executor loss can cause task to not be resubmitted >

[jira] [Resolved] (SPARK-24755) Executor loss can cause task to not be resubmitted

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24755. --- Resolution: Fixed Fix Version/s: 2.3.3 2.4.0 > Executor loss can

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549356#comment-16549356 ] Thomas Graves commented on SPARK-22151: --- ok thanks, must have missed that. > PYTHONPATH not

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549348#comment-16549348 ] Thomas Graves commented on SPARK-22151: --- [~srowen] why did you blank out fixed version and

[jira] [Resolved] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-22151. --- Resolution: Fixed Fix Version/s: 2.4.0 > PYTHONPATH not picked up from the

[jira] [Comment Edited] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549345#comment-16549345 ] Thomas Graves edited comment on SPARK-24615 at 7/19/18 2:28 PM: but my

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549345#comment-16549345 ] Thomas Graves commented on SPARK-24615: --- but my point is exactly that, it shouldn't be yet another

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16549266#comment-16549266 ] Thomas Graves commented on SPARK-24615: --- I think the usage for cpu/memory is the same.  You know

[jira] [Assigned] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-22151: - Assignee: Parth Gandhi > PYTHONPATH not picked up from the spark.yarn.appMasterEnv

[jira] [Updated] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22151: -- Fix Version/s: 2.4.0 > PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly >

[jira] [Updated] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24677: -- Fix Version/s: 2.2.3 > TaskSetManager not updating successfulTaskDurations for old stage

[jira] [Resolved] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24677. --- Resolution: Fixed Fix Version/s: 2.4.0 2.3.3 > TaskSetManager not

[jira] [Assigned] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24677: - Assignee: dzcxzl > TaskSetManager not updating successfulTaskDurations for old stage

[jira] [Updated] (SPARK-24677) TaskSetManager not updating successfulTaskDurations for old stage attempts

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-24677: -- Summary: TaskSetManager not updating successfulTaskDurations for old stage attempts (was:

[jira] [Comment Edited] (SPARK-24677) Avoid NoSuchElementException from MedianHeap

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548210#comment-16548210 ] Thomas Graves edited comment on SPARK-24677 at 7/18/18 6:22 PM: This is

[jira] [Commented] (SPARK-24677) Avoid NoSuchElementException from MedianHeap

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16548210#comment-16548210 ] Thomas Graves commented on SPARK-24677: --- In this case one of the older stage attempts (that is a

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547903#comment-16547903 ] Thomas Graves commented on SPARK-24615: --- did the design doc permissions change? I can't seem to

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546617#comment-16546617 ] Thomas Graves commented on SPARK-24615: --- ok, I agree, I think this SPIP should at least cover how

[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark

2018-07-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546580#comment-16546580 ] Thomas Graves commented on SPARK-24615: --- The user is responsible for asking yarn for the right

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-07-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16545253#comment-16545253 ] Thomas Graves commented on SPARK-24615: --- [~jerryshao] ^ > Accelerator aware task scheduling for

[jira] [Resolved] (SPARK-24610) wholeTextFiles broken for small files

2018-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24610. --- Resolution: Fixed Fix Version/s: 2.4.0 > wholeTextFiles broken for small files >

[jira] [Assigned] (SPARK-24610) wholeTextFiles broken for small files

2018-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24610: - Assignee: Dhruve Ashar > wholeTextFiles broken for small files >

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2018-07-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537035#comment-16537035 ] Thomas Graves commented on SPARK-16534: --- I agree it seems a bit of a bad user story to drop

[jira] [Commented] (SPARK-16534) Kafka 0.10 Python support

2018-07-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16537034#comment-16537034 ] Thomas Graves commented on SPARK-16534: --- If we aren't going to do this we should close this as

[jira] [Updated] (SPARK-13343) speculative tasks that didn't commit shouldn't be marked as success

2018-07-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13343: -- Description: Currently Speculative tasks that didn't commit can show up as success 

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

2018-07-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16529869#comment-16529869 ] Thomas Graves commented on SPARK-17181: --- that would be a question for [~marymwu] > [Spark2.0 web

[jira] [Commented] (SPARK-24615) Accelerator aware task scheduling for Spark

2018-06-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16528293#comment-16528293 ] Thomas Graves commented on SPARK-24615: --- maybe I"m missing it but how is this working with the

[jira] [Commented] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-06-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527865#comment-16527865 ] Thomas Graves commented on SPARK-23309: --- We tried this on a newest 2.3.1 and haven't been able to

[jira] [Resolved] (SPARK-23309) Spark 2.3 cached query performance 20-30% worse then spark 2.2

2018-06-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-23309. --- Resolution: Works for Me > Spark 2.3 cached query performance 20-30% worse then spark 2.2 >

[jira] [Resolved] (SPARK-24372) Create script for preparing RCs

2018-06-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24372. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0 > Create

[jira] [Resolved] (SPARK-24519) MapStatus has 2000 hardcoded

2018-06-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24519. --- Resolution: Fixed Assignee: Hieu Tri Huynh Fix Version/s: 2.4.0 > MapStatus

[jira] [Updated] (SPARK-22897) Expose stageAttemptId in TaskContext

2018-06-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22897: -- Fix Version/s: 2.2.2 > Expose stageAttemptId in TaskContext >

[jira] [Resolved] (SPARK-24589) OutputCommitCoordinator may allow duplicate commits

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24589. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.4.0

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:02 PM: more

[jira] [Comment Edited] (SPARK-24552) Task attempt numbers are reused when stages are retried

2018-06-21 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519445#comment-16519445 ] Thomas Graves edited comment on SPARK-24552 at 6/21/18 3:01 PM: more

<    2   3   4   5   6   7   8   9   10   11   >