[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16309910#comment-16309910 ] Thomas Graves commented on SPARK-22683: --- [~jcuquemelle] . just to confirm your applications

[jira] [Commented] (SPARK-22577) executor page blacklist status should update with TaskSet level blacklisting

[ https://issues.apache.org/jira/browse/SPARK-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298874#comment-16298874 ] Thomas Graves commented on SPARK-22577: --- correct (unfortunately don't time), anyone can take it

[jira] [Commented] (SPARK-22577) executor page blacklist status should update with TaskSet level blacklisting

[ https://issues.apache.org/jira/browse/SPARK-22577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298854#comment-16298854 ] Thomas Graves commented on SPARK-22577: --- Yes I was thinking the stage UI (and perhaps executor UI).

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298589#comment-16298589 ] Thomas Graves commented on SPARK-22765: --- ok so at least based on this one experiment up front

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297549#comment-16297549 ] Thomas Graves commented on SPARK-22765: --- with SPARK-21656 does upfront allocation vs exponential

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297335#comment-16297335 ] Thomas Graves commented on SPARK-22765: --- SPARK-21656 and the dynamic allocation should handle that,

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297307#comment-16297307 ] Thomas Graves commented on SPARK-22765: --- yes between stages becomes a problem with lower timeout,

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296811#comment-16296811 ] Thomas Graves commented on SPARK-22765: --- [~CodingCat] with SPARK-21656 executors shouldn't timeout

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292855#comment-16292855 ] Thomas Graves commented on SPARK-22683: --- Thanks for the clarification, a few of those I misread and

[jira] [Commented] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292608#comment-16292608 ] Thomas Graves commented on SPARK-22465: --- Yes I think that makes sense. > Cogroup of two

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292594#comment-16292594 ] Thomas Graves commented on SPARK-22765: --- I'm not sure how mr style and 4-core executor go together.

[jira] [Commented] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292569#comment-16292569 ] Thomas Graves commented on SPARK-22465: --- I don't have time at the moment to work on this so if you

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291037#comment-16291037 ] Thomas Graves commented on SPARK-22683: --- Another way to approach this is to have a pluggable policy

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289793#comment-16289793 ] Thomas Graves commented on SPARK-22765: --- ok so before you do anything else I would suggest trying

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289700#comment-16289700 ] Thomas Graves commented on SPARK-22765: --- ok so its basically they idle timeout during DAG computing

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289656#comment-16289656 ] Thomas Graves commented on SPARK-22683: --- I am also curious, when you are comparing spark to MR

[jira] [Commented] (SPARK-22683) DynamicAllocation wastes resources by allocating containers that will barely be used

[ https://issues.apache.org/jira/browse/SPARK-22683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289614#comment-16289614 ] Thomas Graves commented on SPARK-22683: --- So the issue brought up here seems to be resource waste vs

[jira] [Commented] (SPARK-22765) Create a new executor allocation scheme based on that of MR

[ https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289613#comment-16289613 ] Thomas Graves commented on SPARK-22765: --- why doesn't idle timeout very small < 5 seconds work? If

[jira] [Commented] (SPARK-22653) executorAddress registered in CoarseGrainedSchedulerBackend.executorDataMap is null

[ https://issues.apache.org/jira/browse/SPARK-22653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270977#comment-16270977 ] Thomas Graves commented on SPARK-22653: --- will have patch up shortly > executorAddress registered

[jira] [Created] (SPARK-22653) executorAddress registered in CoarseGrainedSchedulerBackend.executorDataMap is null

Thomas Graves created SPARK-22653: - Summary: executorAddress registered in CoarseGrainedSchedulerBackend.executorDataMap is null Key: SPARK-22653 URL: https://issues.apache.org/jira/browse/SPARK-22653

[jira] [Commented] (SPARK-22620) Deadlock in blacklisting when executor is removed due to a heartbeat timeout

[ https://issues.apache.org/jira/browse/SPARK-22620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269517#comment-16269517 ] Thomas Graves commented on SPARK-22620: --- Looking at this some more it seems that normally when the

[jira] [Commented] (SPARK-22620) Deadlock in blacklisting when executor is removed due to a heartbeat timeout

[ https://issues.apache.org/jira/browse/SPARK-22620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268792#comment-16268792 ] Thomas Graves commented on SPARK-22620: --- Note this was running spark on yarn in cluster mode, this

[jira] [Created] (SPARK-22620) Deadlock in blacklisting when executor is removed due to a heartbeat timeout

Thomas Graves created SPARK-22620: - Summary: Deadlock in blacklisting when executor is removed due to a heartbeat timeout Key: SPARK-22620 URL: https://issues.apache.org/jira/browse/SPARK-22620

[jira] [Created] (SPARK-22577) executor page blacklist status should update with TaskSet level blacklisting

Thomas Graves created SPARK-22577: - Summary: executor page blacklist status should update with TaskSet level blacklisting Key: SPARK-22577 URL: https://issues.apache.org/jira/browse/SPARK-22577

[jira] [Commented] (SPARK-22465) Cogroup of two disproportionate RDDs could lead into 2G limit BUG

[ https://issues.apache.org/jira/browse/SPARK-22465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16246595#comment-16246595 ] Thomas Graves commented on SPARK-22465: --- Its not strictly the 2G limit. He did hit that but he hit

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198665#comment-16198665 ] Thomas Graves commented on SPARK-21737: --- I haven't had time to think about it much since the pr

[jira] [Updated] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

[ https://issues.apache.org/jira/browse/SPARK-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22218: -- Description: Running on yarn, If you have any application re-attempts using the spark 2.2

[jira] [Commented] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

[ https://issues.apache.org/jira/browse/SPARK-22218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16194803#comment-16194803 ] Thomas Graves commented on SPARK-22218: --- Working on a patch for this. > spark shuffle services

[jira] [Created] (SPARK-22218) spark shuffle services fails to update secret on application re-attempts

Thomas Graves created SPARK-22218: - Summary: spark shuffle services fails to update secret on application re-attempts Key: SPARK-22218 URL: https://issues.apache.org/jira/browse/SPARK-22218 Project:

[jira] [Commented] (SPARK-21737) Create communication channel between arbitrary clients and the Spark AM in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-21737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184120#comment-16184120 ] Thomas Graves commented on SPARK-21737: --- We definitely still want to do this. I don't think

[jira] [Updated] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-22151: -- Description: Running in yarn cluster mode and trying to set pythonpath via

[jira] [Created] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

Thomas Graves created SPARK-22151: - Summary: PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly Key: SPARK-22151 URL: https://issues.apache.org/jira/browse/SPARK-22151 Project: Spark

[jira] [Commented] (SPARK-21157) Report Total Memory Used by Spark Executors

[ https://issues.apache.org/jira/browse/SPARK-21157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179496#comment-16179496 ] Thomas Graves commented on SPARK-21157: --- Just to point out that yarn/mapreduce/tez already have

[jira] [Created] (SPARK-21945) pyspark --py-files doesn't work in yarn client mode

Thomas Graves created SPARK-21945: - Summary: pyspark --py-files doesn't work in yarn client mode Key: SPARK-21945 URL: https://issues.apache.org/jira/browse/SPARK-21945 Project: Spark Issue

[jira] [Resolved] (SPARK-21890) ObtainCredentials does not pass creds to addDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-21890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21890. --- Resolution: Fixed Assignee: Sanket Reddy Fix Version/s: 2.3.0 >

[jira] [Closed] (SPARK-21937) Spark SQL DDL/DML docs non-existent

[ https://issues.apache.org/jira/browse/SPARK-21937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-21937. - Resolution: Duplicate > Spark SQL DDL/DML docs non-existent >

[jira] [Updated] (SPARK-21937) Spark SQL DDL/DML docs non-existent

[ https://issues.apache.org/jira/browse/SPARK-21937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21937: -- Issue Type: Improvement (was: Bug) > Spark SQL DDL/DML docs non-existent >

[jira] [Created] (SPARK-21937) Spark SQL DDL/DML docs non-existent

Thomas Graves created SPARK-21937: - Summary: Spark SQL DDL/DML docs non-existent Key: SPARK-21937 URL: https://issues.apache.org/jira/browse/SPARK-21937 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16154006#comment-16154006 ] Thomas Graves commented on SPARK-21888: --- the client needs to get the hbase credentials for secure

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153744#comment-16153744 ] Thomas Graves commented on SPARK-21888: --- also note that you can do this in client mode by using the

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16153739#comment-16153739 ] Thomas Graves commented on SPARK-21888: --- [~mgaido] I don't think that is true unless something

[jira] [Updated] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21888: -- Description: While running Spark on Yarn in cluster mode, currently there is no way to add any

[jira] [Updated] (SPARK-21890) ObtainCredentials does not pass creds to addDelegationTokens

[ https://issues.apache.org/jira/browse/SPARK-21890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21890: -- Description: I observed this while running a oozie job trying to connect to hbase via spark.

[jira] [Comment Edited] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150530#comment-16150530 ] Thomas Graves edited comment on SPARK-21888 at 9/1/17 1:37 PM: --- Putting

[jira] [Commented] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16150530#comment-16150530 ] Thomas Graves commented on SPARK-21888: --- Putting things into SPARK_CONF_DIR will work, the question

[jira] [Updated] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode

[ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21888: -- Issue Type: Improvement (was: Bug) > Cannot add stuff to Client Classpath for Yarn Cluster

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145413#comment-16145413 ] Thomas Graves commented on SPARK-21841: --- sorry for so many comments, I guess this goes back to

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145399#comment-16145399 ] Thomas Graves commented on SPARK-21841: --- Oh no I guess that is explicit check in spark 2.1: if

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145396#comment-16145396 ] Thomas Graves commented on SPARK-21841: --- Unfortunately that isn't working for me, but might be

[jira] [Comment Edited] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145392#comment-16145392 ] Thomas Graves edited comment on SPARK-21841 at 8/29/17 2:39 PM: I'll try

[jira] [Commented] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-21841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145392#comment-16145392 ] Thomas Graves commented on SPARK-21841: --- I'll try that out. Honestly I didn't even know you could

[jira] [Commented] (SPARK-15689) Data source API v2

[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145304#comment-16145304 ] Thomas Graves commented on SPARK-15689: --- I haven't had a chance to look at this, someone just

[jira] [Resolved] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21798. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0

[jira] [Updated] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21798: -- Issue Type: Bug (was: Improvement) > No config to replace deprecated SPARK_CLASSPATH config

[jira] [Created] (SPARK-21841) Spark SQL doesn't pick up column added in hive when table created with saveAsTable

Thomas Graves created SPARK-21841: - Summary: Spark SQL doesn't pick up column added in hive when table created with saveAsTable Key: SPARK-21841 URL: https://issues.apache.org/jira/browse/SPARK-21841

[jira] [Resolved] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21501. --- Resolution: Fixed Assignee: Sanket Reddy Fix Version/s: 2.3.0 > Spark

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138347#comment-16138347 ] Thomas Graves commented on SPARK-17321: --- Yes that sounds good. It wouldn't hurt to verify the

[jira] [Commented] (SPARK-21798) No config to replace deprecated SPARK_CLASSPATH config for launching daemons like History Server

[ https://issues.apache.org/jira/browse/SPARK-21798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137310#comment-16137310 ] Thomas Graves commented on SPARK-21798: --- You could need to add things to the classpath for custom

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129326#comment-16129326 ] Thomas Graves commented on SPARK-21753: --- I should also state that many other things work fine with

[jira] [Commented] (SPARK-21753) running pi example with pypy on spark fails to serialize

[ https://issues.apache.org/jira/browse/SPARK-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16129320#comment-16129320 ] Thomas Graves commented on SPARK-21753: --- [~holden.ka...@gmail.com] would be curious if you have run

[jira] [Created] (SPARK-21753) running pi example with pypy on spark fails to serialize

Thomas Graves created SPARK-21753: - Summary: running pi example with pypy on spark fails to serialize Key: SPARK-21753 URL: https://issues.apache.org/jira/browse/SPARK-21753 Project: Spark

[jira] [Assigned] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21656: - Assignee: Jong Yoon Lee > spark dynamic allocation should not idle timeout executors

[jira] [Resolved] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21656. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > spark dynamic

[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage

[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127942#comment-16127942 ] Thomas Graves commented on SPARK-20589: --- Note that this type of option is also already supported in

[jira] [Commented] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

[ https://issues.apache.org/jira/browse/SPARK-21714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16127204#comment-16127204 ] Thomas Graves commented on SPARK-21714: --- I haven't had time to get to it, so it would be great if

[jira] [Created] (SPARK-21714) SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again

Thomas Graves created SPARK-21714: - Summary: SparkSubmit in Yarn Client mode downloads remote files and then reuploads them again Key: SPARK-21714 URL: https://issues.apache.org/jira/browse/SPARK-21714

[jira] [Comment Edited] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123439#comment-16123439 ] Thomas Graves edited comment on SPARK-21656 at 8/11/17 2:43 PM: Note,

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123439#comment-16123439 ] Thomas Graves commented on SPARK-21656: --- Note, I've never said there is no counter part

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123435#comment-16123435 ] Thomas Graves commented on SPARK-21656: --- Yes there is a trade off here, use some more resource or

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123393#comment-16123393 ] Thomas Graves commented on SPARK-21656: --- I don't know what you mean by busy driver. The example of

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123358#comment-16123358 ] Thomas Graves commented on SPARK-21656: --- example of test results with this. We have production job

[jira] [Created] (SPARK-21695) Spark scheduler locality algorithm can take longer then expected

Thomas Graves created SPARK-21695: - Summary: Spark scheduler locality algorithm can take longer then expected Key: SPARK-21695 URL: https://issues.apache.org/jira/browse/SPARK-21695 Project: Spark

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Description: Right now with dynamic allocation spark starts by getting the number of

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Description: Right now spark lets go of executors when they are idle for the 60s (or

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when there are enough tasks to run on them

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Summary: spark dynamic allocation should not idle timeout executors when there are enough

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117241#comment-16117241 ] Thomas Graves commented on SPARK-21656: --- As a said above it DOES help the application to keep them

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117204#comment-16117204 ] Thomas Graves commented on SPARK-21656: --- Another option would be just to add logic for spark to

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117200#comment-16117200 ] Thomas Graves commented on SPARK-21656: --- why not fix the bug in dynamic allocation? changing

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117159#comment-16117159 ] Thomas Graves commented on SPARK-21656: --- If given more time the scheduler would have fallen back to

[jira] [Commented] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16117086#comment-16117086 ] Thomas Graves commented on SPARK-21656: --- The executor can be idle if the scheduler doesn't put any

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Issue Type: Bug (was: Improvement) > spark dynamic allocation should not idle timeout

[jira] [Updated] (SPARK-21656) spark dynamic allocation should not idle timeout executors when tasks still to run

[ https://issues.apache.org/jira/browse/SPARK-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21656: -- Priority: Major (was: Minor) > spark dynamic allocation should not idle timeout executors

[jira] [Commented] (SPARK-21655) Kill CLI for Yarn mode

[ https://issues.apache.org/jira/browse/SPARK-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116989#comment-16116989 ] Thomas Graves commented on SPARK-21655: --- The UI kill requests are acl protected. You do need to

[jira] [Commented] (SPARK-21655) Kill CLI for Yarn mode

[ https://issues.apache.org/jira/browse/SPARK-21655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16116859#comment-16116859 ] Thomas Graves commented on SPARK-21655: --- the yarn kill does work, but it does a kill signal on the

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: (was: Nuochen Lyu) > Speculative task that got CommitDenied exception

[jira] [Assigned] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20713: - Assignee: Nuochen Lyu > Speculative task that got CommitDenied exception shows up as

[jira] [Resolved] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed

[ https://issues.apache.org/jira/browse/SPARK-20713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20713. --- Resolution: Fixed Assignee: Nuochen Lyu Fix Version/s: 2.3.0 > Speculative

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108913#comment-16108913 ] Thomas Graves commented on SPARK-21585: --- https://github.com/apache/spark/pull/18788 > Application

[jira] [Resolved] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21585. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Application

[jira] [Commented] (SPARK-21585) Application Master marking application status as Failed for Client Mode

[ https://issues.apache.org/jira/browse/SPARK-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108901#comment-16108901 ] Thomas Graves commented on SPARK-21585: --- [~srowen]do you know if the github pull request link isn't

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105164#comment-16105164 ] Thomas Graves commented on SPARK-17321: --- Can you clarify? as stated above you should not be using

[jira] [Commented] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105133#comment-16105133 ] Thomas Graves commented on SPARK-21541: --- it was merged

[jira] [Resolved] (SPARK-21541) Spark Logs show incorrect job status for a job that does not create SparkContext

[ https://issues.apache.org/jira/browse/SPARK-21541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21541. --- Resolution: Fixed Assignee: Parth Gandhi Fix Version/s: 2.3.0 > Spark Logs

[jira] [Created] (SPARK-21530) Update description of spark.shuffle.maxChunksBeingTransferred

Thomas Graves created SPARK-21530: - Summary: Update description of spark.shuffle.maxChunksBeingTransferred Key: SPARK-21530 URL: https://issues.apache.org/jira/browse/SPARK-21530 Project: Spark

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1616#comment-1616 ] Thomas Graves commented on SPARK-21501: --- We want to change it from a # of entries to a size of

[jira] [Commented] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16098907#comment-16098907 ] Thomas Graves commented on SPARK-21501: --- The issue was actually introduced with SPARK-15074.

[jira] [Updated] (SPARK-21501) Spark shuffle index cache size should be memory based

[ https://issues.apache.org/jira/browse/SPARK-21501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21501: -- Affects Version/s: (was: 2.0.0) 2.1.0 > Spark shuffle index cache

[jira] [Updated] (SPARK-21243) Limit the number of maps in a single shuffle fetch

[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21243: -- Fix Version/s: 2.2.1 > Limit the number of maps in a single shuffle fetch >

[jira] [Created] (SPARK-21501) Spark shuffle index cache size should be memory based

Thomas Graves created SPARK-21501: - Summary: Spark shuffle index cache size should be memory based Key: SPARK-21501 URL: https://issues.apache.org/jira/browse/SPARK-21501 Project: Spark

[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full

[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095546#comment-16095546 ] Thomas Graves commented on SPARK-21460: --- I didn't think that was the case, but took a look at the

<    4   5   6   7   8   9   10   11   12   13   >