[jira] [Closed] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

2016-04-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-3374. Resolution: Duplicate > Spark on Yarn remove deprecated configs for 2.0 > --

[jira] [Commented] (SPARK-3374) Spark on Yarn remove deprecated configs for 2.0

2016-04-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15260492#comment-15260492 ] Thomas Graves commented on SPARK-3374: -- I think these were all removed under SPARK-12

[jira] [Created] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-04-27 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-14963: - Summary: YarnShuffleService should use YARN getRecoveryPath() for leveldb location Key: SPARK-14963 URL: https://issues.apache.org/jira/browse/SPARK-14963 Project:

[jira] [Updated] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-04-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14963: -- Issue Type: Improvement (was: Bug) > YarnShuffleService should use YARN getRecoveryPath() for

[jira] [Resolved] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

2016-04-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-6735. -- Resolution: Fixed Fix Version/s: 2.0.0 > Provide options to make maximum executor failure

[jira] [Updated] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

2016-04-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-6735: - Assignee: Saisai Shao > Provide options to make maximum executor failure count ( which kills the

[jira] [Commented] (SPARK-1989) Exit executors faster if they get into a cycle of heavy GC

2016-04-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15262761#comment-15262761 ] Thomas Graves commented on SPARK-1989: -- Personally I don't agree with this and think

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-04-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264010#comment-15264010 ] Thomas Graves commented on SPARK-14963: --- I'm definitely fine with it but someone el

[jira] [Commented] (SPARK-11316) coalesce doesn't handle UnionRDD with partial locality properly

2016-04-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15264205#comment-15264205 ] Thomas Graves commented on SPARK-11316: --- Simple steps to reproduce an RDD with part

[jira] [Updated] (SPARK-4224) Support group acls

2016-05-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-4224: - Target Version/s: 2.0.0 > Support group acls > -- > > Key: SPARK-4

[jira] [Updated] (SPARK-4224) Support group acls

2016-05-04 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-4224: - Assignee: Dhruve Ashar > Support group acls > -- > > Key: SPARK-42

[jira] [Resolved] (SPARK-4224) Support group acls

2016-05-04 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-4224. -- Resolution: Fixed Fix Version/s: 2.0.0 > Support group acls > -- > >

[jira] [Created] (SPARK-15121) Improve logging of external shuffle handler

2016-05-04 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-15121: - Summary: Improve logging of external shuffle handler Key: SPARK-15121 URL: https://issues.apache.org/jira/browse/SPARK-15121 Project: Spark Issue Type: Imp

[jira] [Updated] (SPARK-15121) Improve logging of external shuffle handler

2016-05-04 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15121: -- Assignee: (was: Thomas Graves) > Improve logging of external shuffle handler >

[jira] [Commented] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-05-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15272466#comment-15272466 ] Thomas Graves commented on SPARK-14963: --- [~jerryshao] I think the other pr is being

[jira] [Created] (SPARK-15178) Remove LazyFileRegion

2016-05-06 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-15178: - Summary: Remove LazyFileRegion Key: SPARK-15178 URL: https://issues.apache.org/jira/browse/SPARK-15178 Project: Spark Issue Type: Improvement Com

[jira] [Updated] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14963: -- Assignee: Saisai Shao > YarnShuffleService should use YARN getRecoveryPath() for leveldb locati

[jira] [Resolved] (SPARK-14963) YarnShuffleService should use YARN getRecoveryPath() for leveldb location

2016-05-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-14963. --- Resolution: Fixed Fix Version/s: 2.1.0 > YarnShuffleService should use YARN getRecover

[jira] [Commented] (SPARK-4924) Factor out code to launch Spark applications into a separate library

2016-05-16 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15284483#comment-15284483 ] Thomas Graves commented on SPARK-4924: -- [~javadba] If you have ideas on improving th

[jira] [Created] (SPARK-15405) YARN uploading the same __spark_conf__.zip twice

2016-05-19 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-15405: - Summary: YARN uploading the same __spark_conf__.zip twice Key: SPARK-15405 URL: https://issues.apache.org/jira/browse/SPARK-15405 Project: Spark Issue Type

[jira] [Commented] (SPARK-15405) YARN uploading the same __spark_conf__.zip twice

2016-05-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15291180#comment-15291180 ] Thomas Graves commented on SPARK-15405: --- [~vanzin] I missed this when I did the re

[jira] [Created] (SPARK-15410) spark-submit --help throws exception

2016-05-19 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-15410: - Summary: spark-submit --help throws exception Key: SPARK-15410 URL: https://issues.apache.org/jira/browse/SPARK-15410 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

2016-05-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296416#comment-15296416 ] Thomas Graves commented on SPARK-14331: --- I was trying to reproduce this to get you

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

2016-05-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296440#comment-15296440 ] Thomas Graves commented on SPARK-14331: --- Note I was running spark on yarn. > Excep

[jira] [Updated] (SPARK-14279) Improve the spark build to pick the version information from the pom file and add git commit information

2016-05-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-14279: -- Assignee: (was: Sanket Reddy) > Improve the spark build to pick the version information fro

[jira] [Commented] (SPARK-14331) Exceptions saving to parquetFile after join from dataframes in master

2016-05-23 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15296705#comment-15296705 ] Thomas Graves commented on SPARK-14331: --- that might be https://github.com/apache/sp

[jira] [Updated] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

2016-05-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13148: -- Summary: document zero-keytab Oozie application launch; add diagnostics (was: support zero-key

[jira] [Updated] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

2016-05-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-13148: -- Assignee: Steve Loughran > document zero-keytab Oozie application launch; add diagnostics > ---

[jira] [Resolved] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

2016-05-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13148. --- Resolution: Fixed Fix Version/s: 2.0.0 > document zero-keytab Oozie application launch

[jira] [Commented] (SPARK-13148) document zero-keytab Oozie application launch; add diagnostics

2016-05-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302687#comment-15302687 ] Thomas Graves commented on SPARK-13148: --- note we changed this to just document what

[jira] [Commented] (SPARK-15671) performance regression CoalesceRDD large # partitions

2016-05-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307934#comment-15307934 ] Thomas Graves commented on SPARK-15671: --- I should have a patch up for this shortly.

[jira] [Created] (SPARK-15671) performance regression CoalesceRDD large # partitions

2016-05-31 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-15671: - Summary: performance regression CoalesceRDD large # partitions Key: SPARK-15671 URL: https://issues.apache.org/jira/browse/SPARK-15671 Project: Spark Issue

[jira] [Updated] (SPARK-15671) performance regression CoalesceRDD large # partitions

2016-05-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15671: -- Target Version/s: 2.0.0 > performance regression CoalesceRDD large # partitions > -

[jira] [Created] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-28 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10858: - Summary: YARN: archives/jar/files rename with # doesn't work unless scheme given Key: SPARK-10858 URL: https://issues.apache.org/jira/browse/SPARK-10858 Project: Sp

[jira] [Comment Edited] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935129#comment-14935129 ] Thomas Graves edited comment on SPARK-10858 at 9/29/15 1:06 PM: ---

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935129#comment-14935129 ] Thomas Graves commented on SPARK-10858: --- yes its a bad thing as users don't know wh

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935134#comment-14935134 ] Thomas Graves commented on SPARK-10858: --- Note the # is the name that we give it on

[jira] [Closed] (SPARK-6508) error handling issue running python in yarn cluster mode

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-6508. Resolution: Duplicate > error handling issue running python in yarn cluster mode >

[jira] [Created] (SPARK-10874) add Search box to History Page

2015-09-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10874: - Summary: add Search box to History Page Key: SPARK-10874 URL: https://issues.apache.org/jira/browse/SPARK-10874 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-10873) can't sort columns on history page

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935371#comment-14935371 ] Thomas Graves commented on SPARK-10873: --- Note if there isn't a jira filed to fix th

[jira] [Commented] (SPARK-10873) can't sort columns on history page

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935370#comment-14935370 ] Thomas Graves commented on SPARK-10873: --- evidentally this was explicitly disabled b

[jira] [Created] (SPARK-10873) can't sort columns on history page

2015-09-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10873: - Summary: can't sort columns on history page Key: SPARK-10873 URL: https://issues.apache.org/jira/browse/SPARK-10873 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-10172) History Server web UI gets messed up when sorting on any column

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935369#comment-14935369 ] Thomas Graves commented on SPARK-10172: --- Was there another jira filed to really fix

[jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935359#comment-14935359 ] Thomas Graves commented on SPARK-6951: -- I'm still seeing history server startup reall

[jira] [Closed] (SPARK-10735) CatalystTypeConverters MatchError converting RDD with custom object to dataframe

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves closed SPARK-10735. - Resolution: Invalid user changed their code. > CatalystTypeConverters MatchError converting RDD

[jira] [Commented] (SPARK-10873) can't sort columns on history page

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935677#comment-14935677 ] Thomas Graves commented on SPARK-10873: --- it might be ok to leave if we implement se

[jira] [Commented] (SPARK-10873) can't sort columns on history page

2015-09-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935688#comment-14935688 ] Thomas Graves commented on SPARK-10873: --- Yeah it should sort all the data also.. >

[jira] [Created] (SPARK-10876) display total application time in spark history UI

2015-09-29 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10876: - Summary: display total application time in spark history UI Key: SPARK-10876 URL: https://issues.apache.org/jira/browse/SPARK-10876 Project: Spark Issue Ty

[jira] [Created] (SPARK-10901) spark.yarn.user.classpath.first doesn't work

2015-10-01 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10901: - Summary: spark.yarn.user.classpath.first doesn't work Key: SPARK-10901 URL: https://issues.apache.org/jira/browse/SPARK-10901 Project: Spark Issue Type: Bu

[jira] [Assigned] (SPARK-10901) spark.yarn.user.classpath.first doesn't work

2015-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-10901: - Assignee: Thomas Graves > spark.yarn.user.classpath.first doesn't work > ---

[jira] [Commented] (SPARK-10901) spark.yarn.user.classpath.first doesn't work

2015-10-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14940264#comment-14940264 ] Thomas Graves commented on SPARK-10901: --- [~vanzin] thanks for the suggestion. Did

[jira] [Created] (SPARK-10910) spark.{executor,driver}.userClassPathFirst don't work for kryo (probably others)

2015-10-02 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10910: - Summary: spark.{executor,driver}.userClassPathFirst don't work for kryo (probably others) Key: SPARK-10910 URL: https://issues.apache.org/jira/browse/SPARK-10910 Pr

[jira] [Commented] (SPARK-10910) spark.{executor,driver}.userClassPathFirst don't work for kryo (probably others)

2015-10-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941209#comment-14941209 ] Thomas Graves commented on SPARK-10910: --- So I understand that perhaps this mechanis

[jira] [Created] (SPARK-10911) Executors should System.exit on clean shutdown

2015-10-02 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10911: - Summary: Executors should System.exit on clean shutdown Key: SPARK-10911 URL: https://issues.apache.org/jira/browse/SPARK-10911 Project: Spark Issue Type:

[jira] [Commented] (SPARK-10911) Executors should System.exit on clean shutdown

2015-10-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14941247#comment-14941247 ] Thomas Graves commented on SPARK-10911: --- Note in this case I was running on Yarn an

[jira] [Created] (SPARK-10930) History "Stages" page "duration" can be confusing

2015-10-05 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10930: - Summary: History "Stages" page "duration" can be confusing Key: SPARK-10930 URL: https://issues.apache.org/jira/browse/SPARK-10930 Project: Spark Issue Typ

[jira] [Created] (SPARK-10971) sparkR: RRunner should allow setting path to Rscript

2015-10-07 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-10971: - Summary: sparkR: RRunner should allow setting path to Rscript Key: SPARK-10971 URL: https://issues.apache.org/jira/browse/SPARK-10971 Project: Spark Issue

[jira] [Assigned] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-10858: - Assignee: Thomas Graves > YARN: archives/jar/files rename with # doesn't work unless sch

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14948941#comment-14948941 ] Thomas Graves commented on SPARK-10858: --- Sorry for the delay on this didn't have ti

[jira] [Commented] (SPARK-10971) sparkR: RRunner should allow setting path to Rscript

2015-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949199#comment-14949199 ] Thomas Graves commented on SPARK-10971: --- you shouldn't have to install everything a

[jira] [Commented] (SPARK-10858) YARN: archives/jar/files rename with # doesn't work unless scheme given

2015-10-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14949334#comment-14949334 ] Thomas Graves commented on SPARK-10858: --- so now I'm really confused why this is bac

[jira] [Commented] (SPARK-10930) History "Stages" page "duration" can be confusing

2015-10-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953469#comment-14953469 ] Thomas Graves commented on SPARK-10930: --- [~pwendell] Can you add Derek as a contri

[jira] [Commented] (SPARK-10930) History "Stages" page "duration" can be confusing

2015-10-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14953486#comment-14953486 ] Thomas Graves commented on SPARK-10930: --- thanks [~srowen] > History "Stages" page

[jira] [Updated] (SPARK-11026) spark.yarn.user.classpath.first does work for 'spark-submit --jars hdfs://user/foo.jar'

2015-10-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11026: -- Summary: spark.yarn.user.classpath.first does work for 'spark-submit --jars hdfs://user/foo.jar

[jira] [Commented] (SPARK-11026) spark.yarn.user.classpath.first does work for 'spark-submit --jars hdfs://user/foo.jar'

2015-10-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14954928#comment-14954928 ] Thomas Graves commented on SPARK-11026: --- if you use --jars hdfs://users/foo.jar wit

[jira] [Resolved] (SPARK-11026) spark.yarn.user.classpath.first does work for 'spark-submit --jars hdfs://user/foo.jar'

2015-10-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-11026. --- Resolution: Fixed Assignee: Lianhui Wang Fix Version/s: 1.6.0

[jira] [Assigned] (SPARK-10619) Can't sort columns on Executor Page

2015-10-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-10619: - Assignee: Thomas Graves > Can't sort columns on Executor Page >

[jira] [Commented] (SPARK-10619) Can't sort columns on Executor Page

2015-10-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14955548#comment-14955548 ] Thomas Graves commented on SPARK-10619: --- looks like this was broken by commit: ht

[jira] [Commented] (SPARK-10873) can't sort columns on history page

2015-10-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957080#comment-14957080 ] Thomas Graves commented on SPARK-10873: --- Has anyone discussed just using something

[jira] [Updated] (SPARK-10876) display total application time in spark history UI

2015-10-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-10876: -- Assignee: Jean-Baptiste Onofré > display total application time in spark history UI > -

[jira] [Commented] (SPARK-10873) can't sort columns on history page

2015-10-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957360#comment-14957360 ] Thomas Graves commented on SPARK-10873: --- [~vanzin] I assume your comment about the

[jira] [Created] (SPARK-11185) Add more task metrics to the "all Stages Page"

2015-10-19 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-11185: - Summary: Add more task metrics to the "all Stages Page" Key: SPARK-11185 URL: https://issues.apache.org/jira/browse/SPARK-11185 Project: Spark Issue Type:

[jira] [Created] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

2015-10-26 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-11316: - Summary: isEmpty before coalesce seems to cause huge performance issue in setupGroups Key: SPARK-11316 URL: https://issues.apache.org/jira/browse/SPARK-11316 Projec

[jira] [Updated] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

2015-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11316: -- Priority: Critical (was: Major) > isEmpty before coalesce seems to cause huge performance issu

[jira] [Commented] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

2015-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14974236#comment-14974236 ] Thomas Graves commented on SPARK-11316: --- Note I"m wondering if since the isEmpty ca

[jira] [Updated] (SPARK-11316) isEmpty before coalesce seems to cause huge performance issue in setupGroups

2015-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11316: -- Description: So I haven't fully debugged this yet but reporting what I'm seeing and think migh

[jira] [Commented] (SPARK-6951) History server slow startup if the event log directory is large

2015-10-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14976364#comment-14976364 ] Thomas Graves commented on SPARK-6951: -- Unfortunately for us I don't think the curren

[jira] [Commented] (SPARK-2089) With YARN, preferredNodeLocalityData isn't honored

2015-10-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979204#comment-14979204 ] Thomas Graves commented on SPARK-2089: -- maybe I'm missing something, why are you sayi

[jira] [Commented] (SPARK-10911) Executors should System.exit on clean shutdown

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993695#comment-14993695 ] Thomas Graves commented on SPARK-10911: --- If the framework or driver is telling it t

[jira] [Created] (SPARK-11555) spark on yarn spark-class --num-workers allows 0 and shouldn't

2015-11-06 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-11555: - Summary: spark on yarn spark-class --num-workers allows 0 and shouldn't Key: SPARK-11555 URL: https://issues.apache.org/jira/browse/SPARK-11555 Project: Spark

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers allows 0 and shouldn't

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993741#comment-14993741 ] Thomas Graves commented on SPARK-11555: --- Note I think this was broken with change f

[jira] [Commented] (SPARK-10911) Executors should System.exit on clean shutdown

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993743#comment-14993743 ] Thomas Graves commented on SPARK-10911: --- You could say its a user problem, but its

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers allows 0 and shouldn't

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993750#comment-14993750 ] Thomas Graves commented on SPARK-11555: --- I should probably rephrase because after l

[jira] [Updated] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11555: -- Summary: spark on yarn spark-class --num-workers doesn't work (was: spark on yarn spark-class

[jira] [Updated] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11555: -- Description: Using the old spark-class and --num-workers interface, --num-workers parameter is

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993765#comment-14993765 ] Thomas Graves commented on SPARK-11555: --- No I'm talking about --num-workers, see th

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993775#comment-14993775 ] Thomas Graves commented on SPARK-11555: --- Note that it is broken with --num-executor

[jira] [Assigned] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-11555: - Assignee: Thomas Graves > spark on yarn spark-class --num-workers doesn't work > ---

[jira] [Commented] (SPARK-11555) spark on yarn spark-class --num-workers doesn't work

2015-11-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14993815#comment-14993815 ] Thomas Graves commented on SPARK-11555: --- To fix this we can simply pass in an optio

[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled

2017-07-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075147#comment-16075147 ] Thomas Graves commented on SPARK-21122: --- I agree with Sean on this. if you are aim

[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read

2017-07-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080340#comment-16080340 ] Thomas Graves commented on SPARK-19659: --- can you be more specific here, do you mean

[jira] [Created] (SPARK-21383) YARN: can allocate to many containers

2017-07-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-21383: - Summary: YARN: can allocate to many containers Key: SPARK-21383 URL: https://issues.apache.org/jira/browse/SPARK-21383 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves commented on SPARK-21383: --- Note we saw this with dynamic allocation o

[jira] [Updated] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21383: -- Summary: YARN can allocate to many executors (was: YARN: can allocate to many containers) > Y

[jira] [Comment Edited] (SPARK-21383) YARN can allocate to many executors

2017-07-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16083321#comment-16083321 ] Thomas Graves edited comment on SPARK-21383 at 7/12/17 2:17 AM: ---

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084075#comment-16084075 ] Thomas Graves commented on SPARK-21376: --- Can you please clarify the title and descr

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16084458#comment-16084458 ] Thomas Graves commented on SPARK-21376: --- so you are referring to the org.apache.spa

[jira] [Resolved] (SPARK-21321) Spark very verbose on shutdown confusing users

2017-07-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21321. --- Resolution: Fixed Assignee: Jong Yoon Lee Fix Version/s: 2.3.0

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2017-07-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091531#comment-16091531 ] Thomas Graves commented on SPARK-15703: --- It should not be breaking dynamic allocati

[jira] [Assigned] (SPARK-21243) Limit the number of maps in a single shuffle fetch

2017-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21243: - Assignee: Dhruve Ashar > Limit the number of maps in a single shuffle fetch > --

<    1   2   3   4   5   6   7   8   9   10   >