[jira] [Resolved] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21585. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Application M

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108913#comment-16108913 ] Thomas Graves commented on SPARK-21585: --- https://github.com/apache/spark/pull/18788

[jira] [Resolved] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20713. --- Resolution: Fixed Assignee: Nuochen Lyu Fix Version/s: 2.3.0 > Speculative ta

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: Nuochen Lyu > Speculative task that got CommitDenied exception shows up as fai

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: (was: Nuochen Lyu) > Speculative task that got CommitDenied exception show

[jira] [Commented] (SPARK-21655) Kill CLI for Yarn mode

[ https://issues.apache.org/jira/browse/SPARK-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116859#comment-16116859 ] Thomas Graves commented on SPARK-21655: --- the yarn kill does work, but it does a kil

[jira] [Commented] (SPARK-21655) Kill CLI for Yarn mode

[ https://issues.apache.org/jira/browse/SPARK-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16116989#comment-16116989 ] Thomas Graves commented on SPARK-21655: --- The UI kill requests are acl protected. Y

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Priority: Major (was: Minor) > spark dynamic allocation should not idle timeout executors when

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Issue Type: Bug (was: Improvement) > spark dynamic allocation should not idle timeout executor

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117086#comment-16117086 ] Thomas Graves commented on SPARK-21656: --- The executor can be idle if the scheduler

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117159#comment-16117159 ] Thomas Graves commented on SPARK-21656: --- If given more time the scheduler would hav

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117200#comment-16117200 ] Thomas Graves commented on SPARK-21656: --- why not fix the bug in dynamic allocation?

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117204#comment-16117204 ] Thomas Graves commented on SPARK-21656: --- Another option would be just to add logic

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117241#comment-16117241 ] Thomas Graves commented on SPARK-21656: --- As a said above it DOES help the applicati

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Summary: spark dynamic allocation should not idle timeout executors when there are enough tasks

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Description: Right now spark lets go of executors when they are idle for the 60s (or configura

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Description: Right now with dynamic allocation spark starts by getting the number of executors

[jira] [Created] (SPARK-21695) Spark scheduler locality algorithm can take longer then expected

Thomas Graves created SPARK-21695: - Summary: Spark scheduler locality algorithm can take longer then expected Key: SPARK-21695 URL: https://issues.apache.org/jira/browse/SPARK-21695 Project: Spark

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123358#comment-16123358 ] Thomas Graves commented on SPARK-21656: --- example of test results with this. We hav

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123393#comment-16123393 ] Thomas Graves commented on SPARK-21656: --- I don't know what you mean by busy driver.

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123435#comment-16123435 ] Thomas Graves commented on SPARK-21656: --- Yes there is a trade off here, use some mo

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123439#comment-16123439 ] Thomas Graves commented on SPARK-21656: --- Note, I've never said there is no counter

[jira] [Comment Edited] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16123439#comment-16123439 ] Thomas Graves edited comment on SPARK-21656 at 8/11/17 2:43 PM: ---

[jira] [Created] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

Thomas Graves created SPARK-21714: - Summary: SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again Key: SPARK-21714 URL: https://issues.apache.org/jira/browse/SPARK-21714

[jira] [Commented] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

[ https://issues.apache.org/jira/browse/SPARK-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127204#comment-16127204 ] Thomas Graves commented on SPARK-21714: --- I haven't had time to get to it, so it wou

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16127942#comment-16127942 ] Thomas Graves commented on SPARK-20589: --- Note that this type of option is also alre

[jira] [Resolved] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21656. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > spark dynamic alloca

[jira] [Assigned] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21656: - Assignee: Jong Yoon Lee > spark dynamic allocation should not idle timeout executors whe

[jira] [Created] (SPARK-21753) running pi example with pypy on spark fails to serialize

Thomas Graves created SPARK-21753: - Summary: running pi example with pypy on spark fails to serialize Key: SPARK-21753 URL: https://issues.apache.org/jira/browse/SPARK-21753 Project: Spark I

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129320#comment-16129320 ] Thomas Graves commented on SPARK-21753: --- [~holden.ka...@gmail.com] would be curious

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16129326#comment-16129326 ] Thomas Graves commented on SPARK-21753: --- I should also state that many other things

[jira] [Commented] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16137310#comment-16137310 ] Thomas Graves commented on SPARK-21798: --- You could need to add things to the classp

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16138347#comment-16138347 ] Thomas Graves commented on SPARK-17321: --- Yes that sounds good. It wouldn't hurt to

[jira] [Resolved] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21501. --- Resolution: Fixed Assignee: Sanket Reddy Fix Version/s: 2.3.0 > Spark shuffle

[jira] [Created] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

Thomas Graves created SPARK-21841: - Summary: Spark SQL doesn't pick up column added in hive when table created with saveAsTable Key: SPARK-21841 URL: https://issues.apache.org/jira/browse/SPARK-21841

[jira] [Updated] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21798: -- Issue Type: Bug (was: Improvement) > No config to replace deprecated SPARK_CLASSPATH config fo

[jira] [Resolved] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21798. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-15689) Data source API v2

[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145304#comment-16145304 ] Thomas Graves commented on SPARK-15689: --- I haven't had a chance to look at this, so

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145392#comment-16145392 ] Thomas Graves commented on SPARK-21841: --- I'll try that out. Honestly I didn't even

[jira] [Comment Edited] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145392#comment-16145392 ] Thomas Graves edited comment on SPARK-21841 at 8/29/17 2:39 PM: ---

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145396#comment-16145396 ] Thomas Graves commented on SPARK-21841: --- Unfortunately that isn't working for me, b

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145399#comment-16145399 ] Thomas Graves commented on SPARK-21841: --- Oh no I guess that is explicit check in sp

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16145413#comment-16145413 ] Thomas Graves commented on SPARK-21841: --- sorry for so many comments, I guess this g

[jira] [Updated] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21888: -- Issue Type: Improvement (was: Bug) > Cannot add stuff to Client Classpath for Yarn Cluster Mod

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150530#comment-16150530 ] Thomas Graves commented on SPARK-21888: --- Putting things into SPARK_CONF_DIR will wo

[jira] [Comment Edited] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150530#comment-16150530 ] Thomas Graves edited comment on SPARK-21888 at 9/1/17 1:37 PM:

[jira] [Updated] (SPARK-21890) ObtainCredentials does not pass creds to addDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-21890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21890: -- Description: I observed this while running a oozie job trying to connect to hbase via spark. It

[jira] [Updated] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21888: -- Description: While running Spark on Yarn in cluster mode, currently there is no way to add any

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153739#comment-16153739 ] Thomas Graves commented on SPARK-21888: --- [~mgaido] I don't think that is true unle

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16153744#comment-16153744 ] Thomas Graves commented on SPARK-21888: --- also note that you can do this in client m

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154006#comment-16154006 ] Thomas Graves commented on SPARK-21888: --- the client needs to get the hbase credenti

[jira] [Updated] (SPARK-21937) Spark SQL DDL/DML docs non-existent

[ https://issues.apache.org/jira/browse/SPARK-21937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21937: -- Issue Type: Improvement (was: Bug) > Spark SQL DDL/DML docs non-existent > ---

[jira] [Created] (SPARK-21937) Spark SQL DDL/DML docs non-existent

Thomas Graves created SPARK-21937: - Summary: Spark SQL DDL/DML docs non-existent Key: SPARK-21937 URL: https://issues.apache.org/jira/browse/SPARK-21937 Project: Spark Issue Type: Bug

[jira] [Closed] (SPARK-21937) Spark SQL DDL/DML docs non-existent

[ https://issues.apache.org/jira/browse/SPARK-21937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-21937. - Resolution: Duplicate > Spark SQL DDL/DML docs non-existent > ---

[jira] [Resolved] (SPARK-21890) ObtainCredentials does not pass creds to addDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-21890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21890. --- Resolution: Fixed Assignee: Sanket Reddy Fix Version/s: 2.3.0 > ObtainCredent

[jira] [Created] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

Thomas Graves created SPARK-21945: - Summary: pyspark --py-files doesn't work in yarn client mode Key: SPARK-21945 URL: https://issues.apache.org/jira/browse/SPARK-21945 Project: Spark Issue T

[jira] [Commented] (SPARK-8425) Add blacklist mechanism for task scheduling

[ https://issues.apache.org/jira/browse/SPARK-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379361#comment-15379361 ] Thomas Graves commented on SPARK-8425: -- Slightly different scenario since its not wit

[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379375#comment-15379375 ] Thomas Graves commented on SPARK-5311: -- We should not be creating the event log dir b

[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379376#comment-15379376 ] Thomas Graves commented on SPARK-5311: -- Note as a follow up, obviously you can config

[jira] [Commented] (SPARK-12661) Drop Python 2.6 support in PySpark

[ https://issues.apache.org/jira/browse/SPARK-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384392#comment-15384392 ] Thomas Graves commented on SPARK-12661: --- I think [~nchammas] is talking about dropp

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384571#comment-15384571 ] Thomas Graves commented on SPARK-15703: --- I think we should do both. event queue si

[jira] [Created] (SPARK-16630) Blacklist a node if executors won't launch on it.

Thomas Graves created SPARK-16630: - Summary: Blacklist a node if executors won't launch on it. Key: SPARK-16630 URL: https://issues.apache.org/jira/browse/SPARK-16630 Project: Spark Issue Typ

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384744#comment-15384744 ] Thomas Graves commented on SPARK-16630: --- Note this is more at the resource manager

[jira] [Updated] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16630: -- Description: On YARN, its possible that a node is messed or misconfigured such that a containe

[jira] [Commented] (SPARK-16650) Improve documentation of spark.task.maxFailures

[ https://issues.apache.org/jira/browse/SPARK-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385868#comment-15385868 ] Thomas Graves commented on SPARK-16650: --- I'll try to put up a patch for this later

[jira] [Created] (SPARK-16650) Improve documentation of spark.task.maxFailures

Thomas Graves created SPARK-16650: - Summary: Improve documentation of spark.task.maxFailures Key: SPARK-16650 URL: https://issues.apache.org/jira/browse/SPARK-16650 Project: Spark Issue Ty

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386235#comment-15386235 ] Thomas Graves commented on SPARK-2666: -- I think eventually adding prestart (MapReduce

[jira] [Resolved] (SPARK-15951) Change Executors Page to use datatables to support sorting columns and searching

[ https://issues.apache.org/jira/browse/SPARK-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15951. --- Resolution: Fixed Fix Version/s: 2.1.0 > Change Executors Page to use datatables to su

[jira] [Updated] (SPARK-15951) Change Executors Page to use datatables to support sorting columns and searching

[ https://issues.apache.org/jira/browse/SPARK-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15951: -- Assignee: Kishor Patil > Change Executors Page to use datatables to support sorting columns and

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386561#comment-15386561 ] Thomas Graves commented on SPARK-2666: -- thanks for the explanation. I guess we would

[jira] [Created] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

Thomas Graves created SPARK-16711: - Summary: YarnShuffleService doesn't re-init properly on YARN rolling upgrade Key: SPARK-16711 URL: https://issues.apache.org/jira/browse/SPARK-16711 Project: Spark

[jira] [Commented] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392135#comment-15392135 ] Thomas Graves commented on SPARK-16711: --- Note that what happens here is the yarnshu

[jira] [Commented] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392145#comment-15392145 ] Thomas Graves commented on SPARK-16711: --- Note this happens when security is on. Th

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Summary: Make ListenerBus event queue size configurable (was: Spark UI doesn't show all tasks

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Issue Type: Improvement (was: Bug) > Make ListenerBus event queue size configurable >

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394271#comment-15394271 ] Thomas Graves commented on SPARK-15703: --- Using this jira to make lsitenerbus config

[jira] [Resolved] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15703. --- Resolution: Fixed Fix Version/s: 2.1.0 > Make ListenerBus event queue size configurabl

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Priority: Minor (was: Critical) > Make ListenerBus event queue size configurable > ---

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Assignee: Dhruve Ashar > Make ListenerBus event queue size configurable > -

[jira] [Commented] (SPARK-16744) spark.yarn.appMasterEnv handling assumes values should be appended

[ https://issues.apache.org/jira/browse/SPARK-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395611#comment-15395611 ] Thomas Graves commented on SPARK-16744: --- Along with this we may consider thinking a

[jira] [Commented] (SPARK-16664) Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.

[ https://issues.apache.org/jira/browse/SPARK-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399168#comment-15399168 ] Thomas Graves commented on SPARK-16664: --- I am out of the office until 8/8. Please c

[jira] [Commented] (SPARK-16158) Support pluggable dynamic allocation heuristics

[ https://issues.apache.org/jira/browse/SPARK-16158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421803#comment-15421803 ] Thomas Graves commented on SPARK-16158: --- seems like an ok idea to me, but you have

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428285#comment-15428285 ] Thomas Graves commented on SPARK-16914: --- Actually with SPARK-14963, the cluster adm

[jira] [Commented] (SPARK-17148) NodeManager exit because of exception “Executor is not registered”

[ https://issues.apache.org/jira/browse/SPARK-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428289#comment-15428289 ] Thomas Graves commented on SPARK-17148: --- If this is causing the nodemanager to die

[jira] [Updated] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16673: -- Assignee: Alex Bozarth > New Executor Page displays columns that used to be conditionally hidde

[jira] [Resolved] (SPARK-11227) Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameservice1

[ https://issues.apache.org/jira/browse/SPARK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-11227. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Spark1.5+ HDFS HA mo

[jira] [Resolved] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16673. --- Resolution: Fixed Fix Version/s: 2.1.0 > New Executor Page displays columns that used

[jira] [Updated] (SPARK-11227) Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameservice1

[ https://issues.apache.org/jira/browse/SPARK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11227: -- Assignee: Kousuke Saruta > Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameserv

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430725#comment-15430725 ] Thomas Graves commented on SPARK-16914: --- that is considered a fatal issue for the n

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430855#comment-15430855 ] Thomas Graves commented on SPARK-17181: --- check your log files to see if you see som

[jira] [Resolved] (SPARK-15083) History Server would OOM due to unlimited TaskUIData in some stages

[ https://issues.apache.org/jira/browse/SPARK-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15083. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.1.0 > History Serve

[jira] [Updated] (SPARK-15083) History Server would OOM due to unlimited TaskUIData in some stages

[ https://issues.apache.org/jira/browse/SPARK-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15083: -- Fix Version/s: 2.0.1 > History Server would OOM due to unlimited TaskUIData in some stages > --

[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449250#comment-15449250 ] Thomas Graves commented on SPARK-17243: --- I agree, there are a ton of ways to improv

[jira] [Resolved] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17243. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.1.0 > Spark 2.0 his

[jira] [Updated] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17243: -- Fix Version/s: 2.0.1 > Spark 2.0 history server summary page gets stuck at "loading history sum

[jira] [Created] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

Thomas Graves created SPARK-17433: - Summary: YarnShuffleService doesn't handle moving credentials levelDb Key: SPARK-17433 URL: https://issues.apache.org/jira/browse/SPARK-17433 Project: Spark

[jira] [Commented] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470693#comment-15470693 ] Thomas Graves commented on SPARK-17433: --- will have path up shortly. > YarnShuffleS

[jira] [Comment Edited] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470693#comment-15470693 ] Thomas Graves edited comment on SPARK-17433 at 9/7/16 1:50 PM:

[jira] [Updated] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16711: -- Fix Version/s: 2.0.1 > YarnShuffleService doesn't re-init properly on YARN rolling upgrade > --

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474557#comment-15474557 ] Thomas Graves commented on SPARK-17321: --- so there are 2 possible things here: 1) Y

<    9   10   11   12   13   14   15   16   17   18   >