[jira] [Updated] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories

2017-03-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19812: -- Summary: YARN shuffle service fails to relocate recovery DB directories (was: YARN shuffle ser

[jira] [Created] (SPARK-19812) YARN shuffle service fix moving recovery DB directories

2017-03-03 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19812: - Summary: YARN shuffle service fix moving recovery DB directories Key: SPARK-19812 URL: https://issues.apache.org/jira/browse/SPARK-19812 Project: Spark Iss

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15892318#comment-15892318 ] Thomas Graves commented on SPARK-18769: --- I definitely understand there is an actual

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-03-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15890629#comment-15890629 ] Thomas Graves commented on SPARK-18769: --- [~yuming] I already made a comment on that

[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled

2017-02-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888430#comment-15888430 ] Thomas Graves commented on SPARK-18769: --- {quote} A little clarification in case the

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2017-02-02 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15849965#comment-15849965 ] Thomas Graves commented on SPARK-18085: --- Are the ui updates just updates to use new

[jira] (SPARK-16742) Kerberos support for Spark on Mesos

2017-01-30 Thread Thomas Graves (JIRA)
Title: Message Title Thomas Graves commented on SPARK-16742

[jira] [Resolved] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow

2017-01-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18750. --- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19179. --- Resolution: Fixed Fix Version/s: 2.2.0 > spark.yarn.access.namenodes description is wr

[jira] [Updated] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19179: -- Assignee: Saisai Shao > spark.yarn.access.namenodes description is wrong >

[jira] [Commented] (KAFKA-4514) Add Codec for ZStandard Compression

2017-01-17 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826105#comment-15826105 ] Thomas Graves commented on KAFKA-4514: -- just for reference, the KIP: https://cwiki.a

[jira] [Created] (SPARK-19179) spark.yarn.access.namenodes description is wrong

2017-01-11 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19179: - Summary: spark.yarn.access.namenodes description is wrong Key: SPARK-19179 URL: https://issues.apache.org/jira/browse/SPARK-19179 Project: Spark Issue Type

[jira] [Resolved] (SPARK-19021) Generailize HDFSCredentialProvider to support non HDFS security FS

2017-01-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19021. --- Resolution: Fixed Assignee: Saisai Shao Fix Version/s: 2.2.0 > Generailize HD

[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)

2017-01-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815078#comment-15815078 ] Thomas Graves commented on SPARK-19143: --- [~mridulm80] You say "we added", you are

[jira] [Commented] (SPARK-19112) add codec for ZStandard

2017-01-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15811935#comment-15811935 ] Thomas Graves commented on SPARK-19112: --- For any sort of Hadoop output formats we g

[jira] [Created] (SPARK-19112) add codec for ZStandard

2017-01-06 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-19112: - Summary: add codec for ZStandard Key: SPARK-19112 URL: https://issues.apache.org/jira/browse/SPARK-19112 Project: Spark Issue Type: Improvement C

[jira] [Resolved] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19033. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > HistoryServer still

[jira] [Updated] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19033: -- Assignee: Saisai Shao > HistoryServer still uses old ACLs even if ACLs are updated > --

[jira] [Commented] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated

2017-01-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15795863#comment-15795863 ] Thomas Graves commented on SPARK-19033: --- this should actually be an improvement to

[jira] [Commented] (KAFKA-4514) Add Codec for ZStandard Compression

2016-12-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15741991#comment-15741991 ] Thomas Graves commented on KAFKA-4514: -- [~dongjin] That is fine, go ahead. > Add Co

[jira] [Commented] (KAFKA-4514) Add Codec for ZStandard Compression

2016-12-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15733627#comment-15733627 ] Thomas Graves commented on KAFKA-4514: -- Ok thanks. I wanted to get it filed to get pe

[jira] [Created] (KAFKA-4514) Add Codec for ZStandard Compression

2016-12-08 Thread Thomas Graves (JIRA)
Thomas Graves created KAFKA-4514: Summary: Add Codec for ZStandard Compression Key: KAFKA-4514 URL: https://issues.apache.org/jira/browse/KAFKA-4514 Project: Kafka Issue Type: Improvement

[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files

2016-12-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726701#comment-15726701 ] Thomas Graves commented on SPARK-18733: --- oh nevermind its looking at lastupdated ti

[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files

2016-12-06 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726671#comment-15726671 ] Thomas Graves commented on SPARK-18733: --- yes looks like a dup but I'm not sure on c

[jira] [Commented] (STORM-2228) KafkaSpout does not replay properly when a topic maps to multiple streams

2016-12-01 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/STORM-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15712228#comment-15712228 ] Thomas Graves commented on STORM-2228: -- Personally I think if its broken and no easy

[jira] [Updated] (SPARK-17843) History Server gives no feedback about the application list being incomplete

2016-11-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17843: -- Fix Version/s: 2.0.3 > History Server gives no feedback about the application list being incomp

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2016-11-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15667226#comment-15667226 ] Thomas Graves commented on SPARK-18085: --- I think at a high level its fine. You men

[jira] [Resolved] (SPARK-17843) History Server gives no feedback about the application list being incomplete

2016-11-11 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17843. --- Resolution: Fixed Assignee: Vinayak Joshi Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18357. --- Resolution: Fixed Fix Version/s: 2.2.0 2.1.0 > YARN --files/--archi

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Assignee: Kishor Patil > YARN --files/--archives broke > - > >

[jira] [Commented] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15647941#comment-15647941 ] Thomas Graves commented on SPARK-18357: --- [~kishorvpatil] can you fix please > YARN

[jira] [Updated] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-18357: -- Priority: Blocker (was: Major) > YARN --files/--archives broke > -

[jira] [Created] (SPARK-18357) YARN --files/--archives broke

2016-11-08 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-18357: - Summary: YARN --files/--archives broke Key: SPARK-18357 URL: https://issues.apache.org/jira/browse/SPARK-18357 Project: Spark Issue Type: Bug Com

[jira] [Created] (STORM-2192) Add a new IAutoCredentials plugin to support SSL files

2016-11-08 Thread Thomas Graves (JIRA)
Thomas Graves created STORM-2192: Summary: Add a new IAutoCredentials plugin to support SSL files Key: STORM-2192 URL: https://issues.apache.org/jira/browse/STORM-2192 Project: Apache Storm I

[jira] [Commented] (STORM-2192) Add a new IAutoCredentials plugin to support SSL files

2016-11-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/STORM-2192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15647815#comment-15647815 ] Thomas Graves commented on STORM-2192: -- I will be working on this and should have a p

[jira] [Resolved] (SPARK-18099) Spark distributed cache should throw exception if same file is specified to dropped in --files --archives

2016-11-03 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-18099. --- Resolution: Fixed Assignee: Kishor Patil Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612086#comment-15612086 ] Thomas Graves commented on SPARK-18085: --- Perhaps we can clarify the title on this j

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15609099#comment-15609099 ] Thomas Graves commented on SPARK-18085: --- Can you expand on is the source of most of

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608760#comment-15608760 ] Thomas Graves commented on SPARK-18085: --- I should also say I don't think we should

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608724#comment-15608724 ] Thomas Graves commented on SPARK-18085: --- Hadn't refreshed and seen your comment abo

[jira] [Commented] (SPARK-18085) Scalability enhancements for the History Server

2016-10-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15608719#comment-15608719 ] Thomas Graves commented on SPARK-18085: --- So there are a few things unclear to me ab

[jira] [Commented] (SPARK-17979) Remove deprecated support for config SPARK_YARN_USER_ENV

2016-10-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585664#comment-15585664 ] Thomas Graves commented on SPARK-17979: --- We already had a jira to remove all the de

[jira] [Commented] (SPARK-15708) Tasks table in Detailed Stage page shows ip instead of hostname under Executor ID/Host

2016-10-18 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15585451#comment-15585451 ] Thomas Graves commented on SPARK-15708: --- [~ajbozarth] It was closed because we wer

[jira] [Resolved] (SPARK-17417) Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

2016-10-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17417. --- Resolution: Fixed Assignee: Dhruve Ashar Fix Version/s: 2.1.0

[jira] [Updated] (SPARK-17417) Fix sorting of part files while reconstructing RDD/partition from checkpointed files.

2016-10-10 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17417: -- Affects Version/s: 1.6.0 2.0.0 > Fix sorting of part files while reconst

[jira] [Resolved] (SPARK-17710) ReplSuite fails with ClassCircularityError in master Maven builds

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17710. --- Resolution: Fixed Assignee: Weiqing Yang Fix Version/s: 2.1.0 > ReplSuite fai

[jira] [Commented] (SPARK-16757) Set up caller context to HDFS and Yarn

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15531056#comment-15531056 ] Thomas Graves commented on SPARK-16757: --- we are seeing issues with ReplSuite after

[jira] [Commented] (PIG-5029) Optimize sort case when data is skewed

2016-09-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/PIG-5029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530065#comment-15530065 ] Thomas Graves commented on PIG-5029: Is the question whether spark supports maps that ar

[jira] [Resolved] (SPARK-16757) Set up caller context to HDFS and Yarn

2016-09-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16757. --- Resolution: Fixed Assignee: Weiqing Yang Fix Version/s: 2.1.0 > Set up caller

[jira] [Commented] (OOZIE-2194) oozie job -kill doesn't work with spark action

2016-09-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/OOZIE-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15506527#comment-15506527 ] Thomas Graves commented on OOZIE-2194: -- Maybe I missed it but doesn't OOZIE-2243 use

[jira] [Commented] (OOZIE-2194) oozie job -kill doesn't work with spark action

2016-09-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/OOZIE-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15506482#comment-15506482 ] Thomas Graves commented on OOZIE-2194: -- application tags should already be supported,

[jira] [Commented] (SPARK-17582) Dead executors shouldn't show in the SparkUI

2016-09-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15503461#comment-15503461 ] Thomas Graves commented on SPARK-17582: --- Yes they are meant to be there, status is

[jira] [Resolved] (SPARK-17511) Dynamic allocation race condition: Containers getting marked failed while releasing

2016-09-14 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17511. --- Resolution: Fixed Assignee: Kishor Patil Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-13 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15487203#comment-15487203 ] Thomas Graves commented on SPARK-17321: --- yes that makes sense and as I stated I thi

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15484748#comment-15484748 ] Thomas Graves commented on SPARK-17321: --- Not sure I follow this comment. So you ar

[jira] [Resolved] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17433. --- Resolution: Fixed Assignee: Thomas Graves Fix Version/s: 2.1.0 > YarnShuffleS

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477061#comment-15477061 ] Thomas Graves commented on SPARK-17321: --- Note that if we want to fix this without y

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-09 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477028#comment-15477028 ] Thomas Graves commented on SPARK-17321: --- it is possible but the point of the recove

[jira] [Commented] (SPARK-17321) YARN shuffle service should use good disk from yarn.nodemanager.local-dirs

2016-09-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15474557#comment-15474557 ] Thomas Graves commented on SPARK-17321: --- so there are 2 possible things here: 1) Y

[jira] [Updated] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

2016-09-08 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16711: -- Fix Version/s: 2.0.1 > YarnShuffleService doesn't re-init properly on YARN rolling upgrade > --

[jira] [Commented] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470693#comment-15470693 ] Thomas Graves commented on SPARK-17433: --- will have path up shortly. > YarnShuffleS

[jira] [Comment Edited] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-07 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15470693#comment-15470693 ] Thomas Graves edited comment on SPARK-17433 at 9/7/16 1:50 PM:

[jira] [Created] (SPARK-17433) YarnShuffleService doesn't handle moving credentials levelDb

2016-09-07 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-17433: - Summary: YarnShuffleService doesn't handle moving credentials levelDb Key: SPARK-17433 URL: https://issues.apache.org/jira/browse/SPARK-17433 Project: Spark

[jira] [Updated] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

2016-08-31 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17243: -- Fix Version/s: 2.0.1 > Spark 2.0 history server summary page gets stuck at "loading history sum

[jira] [Resolved] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

2016-08-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17243. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.1.0 > Spark 2.0 his

[jira] [Commented] (SPARK-17243) Spark 2.0 history server summary page gets stuck at "loading history summary" with 10K+ application history

2016-08-30 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15449250#comment-15449250 ] Thomas Graves commented on SPARK-17243: --- I agree, there are a ton of ways to improv

[jira] [Updated] (SPARK-15083) History Server would OOM due to unlimited TaskUIData in some stages

2016-08-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15083: -- Fix Version/s: 2.0.1 > History Server would OOM due to unlimited TaskUIData in some stages > --

[jira] [Resolved] (SPARK-15083) History Server would OOM due to unlimited TaskUIData in some stages

2016-08-24 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15083. --- Resolution: Fixed Assignee: Alex Bozarth Fix Version/s: 2.1.0 > History Serve

[jira] [Commented] (SPARK-17181) [Spark2.0 web ui]The status of the certain jobs is still displayed as running even if all the stages of this job have already finished

2016-08-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430855#comment-15430855 ] Thomas Graves commented on SPARK-17181: --- check your log files to see if you see som

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

2016-08-22 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15430725#comment-15430725 ] Thomas Graves commented on SPARK-16914: --- that is considered a fatal issue for the n

[jira] [Updated] (SPARK-11227) Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameservice1

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-11227: -- Assignee: Kousuke Saruta > Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameserv

[jira] [Resolved] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-16673. --- Resolution: Fixed Fix Version/s: 2.1.0 > New Executor Page displays columns that used

[jira] [Resolved] (SPARK-11227) Spark1.5+ HDFS HA mode throw java.net.UnknownHostException: nameservice1

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-11227. --- Resolution: Fixed Fix Version/s: 2.1.0 2.0.1 > Spark1.5+ HDFS HA mo

[jira] [Updated] (SPARK-16673) New Executor Page displays columns that used to be conditionally hidden

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16673: -- Assignee: Alex Bozarth > New Executor Page displays columns that used to be conditionally hidde

[jira] [Commented] (SPARK-17148) NodeManager exit because of exception “Executor is not registered”

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428289#comment-15428289 ] Thomas Graves commented on SPARK-17148: --- If this is causing the nodemanager to die

[jira] [Commented] (SPARK-16914) NodeManager crash when spark are registering executor infomartion into leveldb

2016-08-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15428285#comment-15428285 ] Thomas Graves commented on SPARK-16914: --- Actually with SPARK-14963, the cluster adm

[jira] [Commented] (SPARK-16158) Support pluggable dynamic allocation heuristics

2016-08-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15421803#comment-15421803 ] Thomas Graves commented on SPARK-16158: --- seems like an ok idea to me, but you have

[jira] [Commented] (SPARK-16664) Spark 1.6.2 - Persist call on Data frames with more than 200 columns is wiping out the data.

2016-07-29 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399168#comment-15399168 ] Thomas Graves commented on SPARK-16664: --- I am out of the office until 8/8. Please c

[jira] [Commented] (SPARK-16744) spark.yarn.appMasterEnv handling assumes values should be appended

2016-07-27 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15395611#comment-15395611 ] Thomas Graves commented on SPARK-16744: --- Along with this we may consider thinking a

[jira] [Resolved] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15703. --- Resolution: Fixed Fix Version/s: 2.1.0 > Make ListenerBus event queue size configurabl

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Priority: Minor (was: Critical) > Make ListenerBus event queue size configurable > ---

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Issue Type: Improvement (was: Bug) > Make ListenerBus event queue size configurable >

[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394271#comment-15394271 ] Thomas Graves commented on SPARK-15703: --- Using this jira to make lsitenerbus config

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Assignee: Dhruve Ashar > Make ListenerBus event queue size configurable > -

[jira] [Updated] (SPARK-15703) Make ListenerBus event queue size configurable

2016-07-26 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15703: -- Summary: Make ListenerBus event queue size configurable (was: Spark UI doesn't show all tasks

[jira] [Commented] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

2016-07-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392145#comment-15392145 ] Thomas Graves commented on SPARK-16711: --- Note this happens when security is on. Th

[jira] [Commented] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

2016-07-25 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392135#comment-15392135 ] Thomas Graves commented on SPARK-16711: --- Note that what happens here is the yarnshu

[jira] [Created] (SPARK-16711) YarnShuffleService doesn't re-init properly on YARN rolling upgrade

2016-07-25 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-16711: - Summary: YarnShuffleService doesn't re-init properly on YARN rolling upgrade Key: SPARK-16711 URL: https://issues.apache.org/jira/browse/SPARK-16711 Project: Spark

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

2016-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386561#comment-15386561 ] Thomas Graves commented on SPARK-2666: -- thanks for the explanation. I guess we would

[jira] [Updated] (SPARK-15951) Change Executors Page to use datatables to support sorting columns and searching

2016-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-15951: -- Assignee: Kishor Patil > Change Executors Page to use datatables to support sorting columns and

[jira] [Resolved] (SPARK-15951) Change Executors Page to use datatables to support sorting columns and searching

2016-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-15951. --- Resolution: Fixed Fix Version/s: 2.1.0 > Change Executors Page to use datatables to su

[jira] [Commented] (SPARK-2666) Always try to cancel running tasks when a stage is marked as zombie

2016-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386235#comment-15386235 ] Thomas Graves commented on SPARK-2666: -- I think eventually adding prestart (MapReduce

[jira] [Created] (SPARK-16650) Improve documentation of spark.task.maxFailures

2016-07-20 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-16650: - Summary: Improve documentation of spark.task.maxFailures Key: SPARK-16650 URL: https://issues.apache.org/jira/browse/SPARK-16650 Project: Spark Issue Ty

[jira] [Commented] (SPARK-16650) Improve documentation of spark.task.maxFailures

2016-07-20 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385868#comment-15385868 ] Thomas Graves commented on SPARK-16650: --- I'll try to put up a patch for this later

[jira] [Updated] (SPARK-16630) Blacklist a node if executors won't launch on it.

2016-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-16630: -- Description: On YARN, its possible that a node is messed or misconfigured such that a containe

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

2016-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384744#comment-15384744 ] Thomas Graves commented on SPARK-16630: --- Note this is more at the resource manager

[jira] [Created] (SPARK-16630) Blacklist a node if executors won't launch on it.

2016-07-19 Thread Thomas Graves (JIRA)
Thomas Graves created SPARK-16630: - Summary: Blacklist a node if executors won't launch on it. Key: SPARK-16630 URL: https://issues.apache.org/jira/browse/SPARK-16630 Project: Spark Issue Typ

[jira] [Commented] (SPARK-15703) Spark UI doesn't show all tasks as completed when it should

2016-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384571#comment-15384571 ] Thomas Graves commented on SPARK-15703: --- I think we should do both. event queue si

[jira] [Commented] (SPARK-12661) Drop Python 2.6 support in PySpark

2016-07-19 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384392#comment-15384392 ] Thomas Graves commented on SPARK-12661: --- I think [~nchammas] is talking about dropp

[jira] [Commented] (SPARK-5311) EventLoggingListener throws exception if log directory does not exist

2016-07-15 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15379376#comment-15379376 ] Thomas Graves commented on SPARK-5311: -- Note as a follow up, obviously you can config

<    6   7   8   9   10   11   12   13   14   15   >