[jira] [Created] (SPARK-4167) Schedule task on Executor will be Imbalance while task run less than local-wait time

2014-10-31 Thread SuYan (JIRA)
SuYan created SPARK-4167: Summary: Schedule task on Executor will be Imbalance while task run less than local-wait time Key: SPARK-4167 URL: https://issues.apache.org/jira/browse/SPARK-4167 Project: Spark

[jira] [Updated] (SPARK-4167) Schedule task on Executor will be Imbalance while task run less than local-wait time

2014-10-31 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-4167: - Description: Recently, when run a spark on yarn job. it occurs executor schedules imbalance. the procedure is

[jira] [Updated] (SPARK-4167) Schedule task on Executor will be Imbalance while task run less than local-wait time

2014-10-31 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-4167: - Description: Recently, when run a spark on yarn job. it occurs executor schedules imbalance. the procedure is

[jira] [Closed] (SPARK-4167) Schedule task on Executor will be Imbalance while task run less than local-wait time

2014-10-31 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan closed SPARK-4167. Resolution: Not a Problem Schedule task on Executor will be Imbalance while task run less than local-wait time

[jira] [Created] (SPARK-4200) akka.loglevel

2014-11-02 Thread SuYan (JIRA)
SuYan created SPARK-4200: Summary: akka.loglevel Key: SPARK-4200 URL: https://issues.apache.org/jira/browse/SPARK-4200 Project: Spark Issue Type: Question Components: Spark Core

[jira] [Created] (SPARK-4471) blockManagerIdFromJson function throws exception while BlockManagerId be null in MetadataFetchFailedException

2014-11-18 Thread SuYan (JIRA)
SuYan created SPARK-4471: Summary: blockManagerIdFromJson function throws exception while BlockManagerId be null in MetadataFetchFailedException Key: SPARK-4471 URL: https://issues.apache.org/jira/browse/SPARK-4471

[jira] [Created] (SPARK-4714) Checking block is null or not after having gotten info.lock in remove block method

2014-12-02 Thread SuYan (JIRA)
SuYan created SPARK-4714: Summary: Checking block is null or not after having gotten info.lock in remove block method Key: SPARK-4714 URL: https://issues.apache.org/jira/browse/SPARK-4714 Project: Spark

[jira] [Created] (SPARK-4721) Improve first thread to put block failed

2014-12-03 Thread SuYan (JIRA)
SuYan created SPARK-4721: Summary: Improve first thread to put block failed Key: SPARK-4721 URL: https://issues.apache.org/jira/browse/SPARK-4721 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-4777) Some block memory after unrollSafely not count into used memory(memoryStore.entrys or unrollMemory)

2014-12-06 Thread SuYan (JIRA)
SuYan created SPARK-4777: Summary: Some block memory after unrollSafely not count into used memory(memoryStore.entrys or unrollMemory) Key: SPARK-4777 URL: https://issues.apache.org/jira/browse/SPARK-4777

[jira] [Commented] (SPARK-4777) Some block memory after unrollSafely not count into used memory(memoryStore.entrys or unrollMemory)

2014-12-16 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249330#comment-14249330 ] SuYan commented on SPARK-4777: -- Sean Owen, Hi, I intended to close that patch, but after

[jira] [Updated] (SPARK-5259) Add task equal() and hashcode() to avoid stage.pendingTasks not accurate while stage was retry

2015-01-14 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-5259: - Description: 1. while shuffle stage was retry, there may have 2 taskSet running. we call the 2

[jira] [Created] (SPARK-5259) Add task equal() and hashcode() to avoid stage.pendingTasks not accurate while stage was retry

2015-01-14 Thread SuYan (JIRA)
SuYan created SPARK-5259: Summary: Add task equal() and hashcode() to avoid stage.pendingTasks not accurate while stage was retry Key: SPARK-5259 URL: https://issues.apache.org/jira/browse/SPARK-5259

[jira] [Updated] (SPARK-5259) Add task equal() and hashcode() to avoid stage.pendingTasks not accurate while stage was retry

2015-01-14 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-5259: - Description: 1. while shuffle stage was retry, there may have 2 taskSet running. we call the 2

[jira] [Updated] (SPARK-5259) Fix endless retry stage by add task equal() and hashcode() to avoid stage.pendingTasks not empty while stage map output is available

2015-01-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-5259: - Summary: Fix endless retry stage by add task equal() and hashcode() to avoid stage.pendingTasks not empty while

[jira] [Created] (SPARK-5132) The name for get stage info atempt ID from Json was wrong

2015-01-07 Thread SuYan (JIRA)
SuYan created SPARK-5132: Summary: The name for get stage info atempt ID from Json was wrong Key: SPARK-5132 URL: https://issues.apache.org/jira/browse/SPARK-5132 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-6606) Accumulator deserialized twice because the NarrowCoGroupSplitDep contains rdd object.

2015-03-30 Thread SuYan (JIRA)
SuYan created SPARK-6606: Summary: Accumulator deserialized twice because the NarrowCoGroupSplitDep contains rdd object. Key: SPARK-6606 URL: https://issues.apache.org/jira/browse/SPARK-6606 Project: Spark

[jira] [Closed] (SPARK-6606) Accumulator deserialized twice because the NarrowCoGroupSplitDep contains rdd object.

2015-03-30 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan closed SPARK-6606. Resolution: Duplicate Duplicate with SPARK-5360, see https://github.com/apache/spark/pull/4145 Accumulator

[jira] [Commented] (SPARK-5945) Spark should not retry a stage infinitely on a FetchFailedException

2015-02-27 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341285#comment-14341285 ] SuYan commented on SPARK-5945: -- I encounter stage retry infinitely when a executor lost

[jira] [Comment Edited] (SPARK-6156) Not cache in memory again if put memory_and_disk level block after put it in disk after unroll unsuccess in memory.

2015-03-04 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346818#comment-14346818 ] SuYan edited comment on SPARK-6156 at 3/4/15 12:08 PM: --- Sean Owen,

[jira] [Created] (SPARK-6157) Unroll unsuccessful memory_and_disk level block should release reserved unroll memory after put success in disk

2015-03-04 Thread SuYan (JIRA)
SuYan created SPARK-6157: Summary: Unroll unsuccessful memory_and_disk level block should release reserved unroll memory after put success in disk Key: SPARK-6157 URL: https://issues.apache.org/jira/browse/SPARK-6157

[jira] [Updated] (SPARK-6156) Refine Put Memory_And_Disk block

2015-03-04 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-6156: - Affects Version/s: 1.2.1 Fix Version/s: (was: 1.3.0) Refine Put Memory_And_Disk block

[jira] [Updated] (SPARK-6157) Unroll unsuccessful memory_and_disk level block should release reserved unroll memory after put success in disk

2015-03-04 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-6157: - Component/s: (was: Spark Core) Block Manager Unroll unsuccessful memory_and_disk level

[jira] [Created] (SPARK-6156) Refine Put Memory_And_Disk block

2015-03-04 Thread SuYan (JIRA)
SuYan created SPARK-6156: Summary: Refine Put Memory_And_Disk block Key: SPARK-6156 URL: https://issues.apache.org/jira/browse/SPARK-6156 Project: Spark Issue Type: Bug Reporter: SuYan

[jira] [Updated] (SPARK-6156) Refine Put Memory_And_Disk block

2015-03-04 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-6156: - Fix Version/s: 1.3.0 Refine Put Memory_And_Disk block Key:

[jira] [Updated] (SPARK-6156) Refine Put Memory_And_Disk block

2015-03-04 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-6156: - Component/s: Spark Core Refine Put Memory_And_Disk block

[jira] [Updated] (SPARK-6156) Not cache in memory again if put memory_and_disk level block after put it in disk after unroll unsuccess in memory.

2015-03-04 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-6156: - Summary: Not cache in memory again if put memory_and_disk level block after put it in disk after unroll unsuccess

[jira] [Commented] (SPARK-8101) Upgrade netty to avoid memory leak accord to netty #3837 issues

2015-06-09 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578839#comment-14578839 ] SuYan commented on SPARK-8101: -- Sorry for late to see that, that problem was fix in

[jira] [Created] (SPARK-8100) Make able to refer lost executor log

2015-06-04 Thread SuYan (JIRA)
SuYan created SPARK-8100: Summary: Make able to refer lost executor log Key: SPARK-8100 URL: https://issues.apache.org/jira/browse/SPARK-8100 Project: Spark Issue Type: Improvement Affects

[jira] [Created] (SPARK-8101) Upgrade netty to avoid memory leak accord to netty #3837 issues

2015-06-04 Thread SuYan (JIRA)
SuYan created SPARK-8101: Summary: Upgrade netty to avoid memory leak accord to netty #3837 issues Key: SPARK-8101 URL: https://issues.apache.org/jira/browse/SPARK-8101 Project: Spark Issue Type:

[jira] [Commented] (SPARK-8101) Upgrade netty to avoid memory leak accord to netty #3837 issues

2015-06-04 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14572715#comment-14572715 ] SuYan commented on SPARK-8101: -- May upgrade in some time Upgrade netty to avoid memory leak

[jira] [Created] (SPARK-8044) Invoid use directMemory while put or get block from file

2015-06-02 Thread SuYan (JIRA)
SuYan created SPARK-8044: Summary: Invoid use directMemory while put or get block from file Key: SPARK-8044 URL: https://issues.apache.org/jira/browse/SPARK-8044 Project: Spark Issue Type:

[jira] [Updated] (SPARK-8044) Avoid to use directMemory while put or get disk level block from file

2015-06-02 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-8044: - Summary: Avoid to use directMemory while put or get disk level block from file (was: Invoid use directMemory

[jira] [Updated] (SPARK-8044) Avoid to use directMemory while put or get disk level block from file

2015-06-02 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-8044: - Description: 1. I found if we use getChannel to put or get data, it will create DirectBuffer anyway, which is

[jira] [Updated] (SPARK-8044) Invoid use directMemory while put or get disk level block from file

2015-06-02 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-8044: - Summary: Invoid use directMemory while put or get disk level block from file (was: Invoid use directMemory while

[jira] [Commented] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-07-02 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612786#comment-14612786 ] SuYan commented on SPARK-5594: -- Do you write sth like: object XXX { val sc = new

[jira] [Commented] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-07-02 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612785#comment-14612785 ] SuYan commented on SPARK-5594: -- Do you write sth like: object XXX { val sc = new

[jira] [Issue Comment Deleted] (SPARK-5594) SparkException: Failed to get broadcast (TorrentBroadcast)

2015-07-02 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-5594: - Comment: was deleted (was: Do you write sth like: object XXX { val sc = new SparkContext() def main {

[jira] [Created] (SPARK-10052) KafKaDirectDstream should filter empty partition task or rdd

2015-08-17 Thread SuYan (JIRA)
SuYan created SPARK-10052: - Summary: KafKaDirectDstream should filter empty partition task or rdd Key: SPARK-10052 URL: https://issues.apache.org/jira/browse/SPARK-10052 Project: Spark Issue Type:

[jira] [Created] (SPARK-11746) Use cache-aware method 'dependencies' to instead of 'getDependencies'

2015-11-15 Thread SuYan (JIRA)
SuYan created SPARK-11746: - Summary: Use cache-aware method 'dependencies' to instead of 'getDependencies' Key: SPARK-11746 URL: https://issues.apache.org/jira/browse/SPARK-11746 Project: Spark

[jira] [Updated] (SPARK-10842) Eliminate create duplicate stage while generate job dag

2015-09-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10842: -- Affects Version/s: 1.5.0 Priority: Minor (was: Major) Description: When we traverse

[jira] [Created] (SPARK-10842) liminate duplicate stage

2015-09-26 Thread SuYan (JIRA)
SuYan created SPARK-10842: - Summary: liminate duplicate stage Key: SPARK-10842 URL: https://issues.apache.org/jira/browse/SPARK-10842 Project: Spark Issue Type: Improvement Reporter:

[jira] [Updated] (SPARK-10842) Eliminate create duplicate stage while generate job dag

2015-09-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10842: -- Description: When we traverse RDD, to generate Stage DAG, Spark will skip to judge the stage whether was

[jira] [Comment Edited] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-27 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909990#comment-14909990 ] SuYan edited comment on SPARK-10796 at 9/28/15 2:59 AM: Running Stage 0, running

[jira] [Commented] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-27 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909990#comment-14909990 ] SuYan commented on SPARK-10796: --- Running Stage 0, running TaskSet0.0, Finshed task0.0 in ExecA, running

[jira] [Comment Edited] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-27 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14909990#comment-14909990 ] SuYan edited comment on SPARK-10796 at 9/28/15 3:00 AM: Running Stage 0.0,

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-28 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Affects Version/s: 1.4.0 1.5.0 > The Stage taskSets may are all removed while stage

[jira] [Commented] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-28 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14933176#comment-14933176 ] SuYan commented on SPARK-10796: --- [~sowen] Hi, I had reproduced that problem in latest version. already

[jira] [Created] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-24 Thread SuYan (JIRA)
SuYan created SPARK-10796: - Summary: The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors Key: SPARK-10796 URL:

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-24 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Description: We meet that problem in Spark 1.3.0, and I also check the latest Spark code, and I think that

[jira] [Comment Edited] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-24 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907532#comment-14907532 ] SuYan edited comment on SPARK-10796 at 9/25/15 4:07 AM: I already refine that

[jira] [Comment Edited] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-24 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907532#comment-14907532 ] SuYan edited comment on SPARK-10796 at 9/25/15 5:11 AM: I already refine that

[jira] [Commented] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-24 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14907532#comment-14907532 ] SuYan commented on SPARK-10796: --- I already refine that description. Simple Example will be add lately. If

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2015-09-24 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Description: We meet that problem in Spark 1.3.0, and I also check the latest Spark code, and I think that

[jira] [Created] (SPARK-12419) FetchFailed = false Executor lost should not allowed re-registered in BlockManager Master again?

2015-12-17 Thread SuYan (JIRA)
SuYan created SPARK-12419: - Summary: FetchFailed = false Executor lost should not allowed re-registered in BlockManager Master again? Key: SPARK-12419 URL: https://issues.apache.org/jira/browse/SPARK-12419

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029427#comment-15029427 ] SuYan commented on SPARK-12009: --- run on the spark 1.4.0, and check current 1.5.2, that problem still exist,

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029410#comment-15029410 ] SuYan commented on SPARK-12009: --- = =, the log is based 1.4.0 {code} override def

[jira] [Updated] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-12009: -- Description: Log based 1.4.0 2015-11-26,03:05:16,176 WARN

[jira] [Created] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-25 Thread SuYan (JIRA)
SuYan created SPARK-12009: - Summary: Avoid re-allocate yarn container while driver want to stop all Executors Key: SPARK-12009 URL: https://issues.apache.org/jira/browse/SPARK-12009 Project: Spark

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-25 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028288#comment-15028288 ] SuYan commented on SPARK-12009: --- user had called sc.stop in main Program > Avoid re-allocate yarn

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-29 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031298#comment-15031298 ] SuYan edited comment on SPARK-12009 at 11/30/15 3:59 AM: - I still think it is

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-29 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031297#comment-15031297 ] SuYan commented on SPARK-12009: --- default time is 10 min...after 10 min, yarn will mark AM as expired, and

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-29 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031283#comment-15031283 ] SuYan commented on SPARK-12009: --- [~jerryshao] I would take some time to look into if Yarn lost heart beat

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-29 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031298#comment-15031298 ] SuYan commented on SPARK-12009: --- I still think it is better to only stop to request new containers > Avoid

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028363#comment-15028363 ] SuYan commented on SPARK-12009: --- AM is not exit, it will exit while driver execute its usercode in

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028363#comment-15028363 ] SuYan edited comment on SPARK-12009 at 11/26/15 8:42 AM: - AM is not exit, it will

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

2015-11-26 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028363#comment-15028363 ] SuYan edited comment on SPARK-12009 at 11/26/15 8:42 AM: - AM is not exit, it will

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-06-08 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15320131#comment-15320131 ] SuYan commented on SPARK-15815: --- Got stage-partition blacklist executors, to found weather the task can run

[jira] [Issue Comment Deleted] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-06-08 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-15815: -- Comment: was deleted (was: Got stage-partition blacklist executors, to found weather the task can run success

[jira] [Created] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-06-08 Thread SuYan (JIRA)
SuYan created SPARK-15815: - Summary: Hang while enable blacklistExecutor and DynamicExecutorAllocator Key: SPARK-15815 URL: https://issues.apache.org/jira/browse/SPARK-15815 Project: Spark Issue

[jira] [Comment Edited] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-06-15 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331285#comment-15331285 ] SuYan edited comment on SPARK-15815 at 6/15/16 7:17 AM: I see... although it can

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-06-15 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15331285#comment-15331285 ] SuYan commented on SPARK-15815: --- I see... although it can solve the gang problem, but for Dynamic Allocate,

[jira] [Commented] (SPARK-12757) Use reference counting to prevent blocks from being evicted during reads

2016-06-02 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15311938#comment-15311938 ] SuYan commented on SPARK-12757: --- [~joshrosen] Hi, can someone do some works to merge this patch and

[jira] [Comment Edited] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-06-16 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333198#comment-15333198 ] SuYan edited comment on SPARK-15815 at 6/16/16 6:24 AM: eh...yes, still have the

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-06-16 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15333198#comment-15333198 ] SuYan commented on SPARK-15815: --- eh...yes, still have the uncertainty to got another executors, how can we

[jira] [Updated] (SPARK-13060) CoarsedExecutorBackend register to driver should wait Executor was ready?

2016-01-28 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-13060: -- Description: Hi Josh I am spark user, currently I feel confused about executor registration. {code} Why

[jira] [Created] (SPARK-13060) CoarsedExecutorBackend register to driver should wait Executor was ready?

2016-01-28 Thread SuYan (JIRA)
SuYan created SPARK-13060: - Summary: CoarsedExecutorBackend register to driver should wait Executor was ready? Key: SPARK-13060 URL: https://issues.apache.org/jira/browse/SPARK-13060 Project: Spark

[jira] [Created] (SPARK-13112) CoarsedExecutorBackend register to driver should wait Executor was ready

2016-02-01 Thread SuYan (JIRA)
SuYan created SPARK-13112: - Summary: CoarsedExecutorBackend register to driver should wait Executor was ready Key: SPARK-13112 URL: https://issues.apache.org/jira/browse/SPARK-13112 Project: Spark

[jira] [Updated] (SPARK-14957) can't connect to Yarn Shuffle service due to it adopt the non-exists dir to store executors metas

2016-04-27 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-14957: -- Description: It always adopt the first dir, and never test if the dir is exsit or can read or can write?

[jira] [Created] (SPARK-14957) can't connect to Yarn Shuffle service due to it adopt the non-exists dir to store executors metas

2016-04-27 Thread SuYan (JIRA)
SuYan created SPARK-14957: - Summary: can't connect to Yarn Shuffle service due to it adopt the non-exists dir to store executors metas Key: SPARK-14957 URL: https://issues.apache.org/jira/browse/SPARK-14957

[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15253237#comment-15253237 ] SuYan commented on SPARK-14750: --- yarn.log-aggregation-enable true // if this =true, means

[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs

2016-05-11 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280044#comment-15280044 ] SuYan commented on SPARK-14750: --- [~vanzin], I just see you comment today, enable MR JobHistoryServer

[jira] [Comment Edited] (SPARK-14750) Make historyServer refer application log in hdfs

2016-05-11 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281144#comment-15281144 ] SuYan edited comment on SPARK-14750 at 5/12/16 3:08 AM: [~vanzin] ah, Thanks to

[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs

2016-05-11 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281144#comment-15281144 ] SuYan commented on SPARK-14750: --- ah, Thanks to help me find a simple way to do this... > Make

[jira] [Commented] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2016-05-10 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15278124#comment-15278124 ] SuYan commented on SPARK-10796: --- main changes: 1. make DAGScheuler only receive Task Resubmit events from

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2016-05-10 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Description: {code} test("Resubmit stage while lost partition in ZombieTasksets or RemovedTaskSets") {

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2016-05-10 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Description: desc: 1. We know a running ShuffleMapStage will have multiple TaskSet: one Active TaskSet,

[jira] [Updated] (SPARK-10796) The Stage taskSets may are all removed while stage still have pending partitions after having lost some executors

2016-05-10 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-10796: -- Description: We meet that problem in Spark 1.3.0, and I also check the latest Spark code, and I think that

[jira] [Created] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-20 Thread SuYan (JIRA)
SuYan created SPARK-14750: - Summary: Make historyServer refer application log in hdfs Key: SPARK-14750 URL: https://issues.apache.org/jira/browse/SPARK-14750 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-20 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251184#comment-15251184 ] SuYan commented on SPARK-14750: --- In yarn mode, spark user can't refer executor log on historyServer logUrl,

[jira] [Created] (SPARK-14804) Graph vertexRDD/EdgeRDD checkpoint results ClassCastException:

2016-04-21 Thread SuYan (JIRA)
SuYan created SPARK-14804: - Summary: Graph vertexRDD/EdgeRDD checkpoint results ClassCastException: Key: SPARK-14804 URL: https://issues.apache.org/jira/browse/SPARK-14804 Project: Spark Issue

[jira] [Updated] (SPARK-14804) Graph vertexRDD/EdgeRDD checkpoint results ClassCastException:

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-14804: -- Description: {code} graph3.vertices.checkpoint() graph3.vertices.count()

[jira] [Updated] (SPARK-14804) Graph vertexRDD/EdgeRDD checkpoint results ClassCastException:

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-14804: -- Priority: Minor (was: Major) > Graph vertexRDD/EdgeRDD checkpoint results ClassCastException: >

[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251608#comment-15251608 ] SuYan commented on SPARK-14750: --- # but there's not a reason to expect they also remain wherever they were

[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251607#comment-15251607 ] SuYan commented on SPARK-14750: --- # but there's not a reason to expect they also remain wherever they were

[jira] [Issue Comment Deleted] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] SuYan updated SPARK-14750: -- Comment: was deleted (was: # but there's not a reason to expect they also remain wherever they were logged

[jira] [Comment Edited] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251558#comment-15251558 ] SuYan edited comment on SPARK-14750 at 4/21/16 8:37 AM: historyServer for spark

[jira] [Commented] (SPARK-14750) Make historyServer refer application log in hdfs

2016-04-21 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15251558#comment-15251558 ] SuYan commented on SPARK-14750: --- historyServer for spark on yarn, the logUrl was something like:

[jira] [Commented] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-07-28 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397271#comment-15397271 ] SuYan commented on SPARK-15815: --- Current temp solution is when all executor were 60s time-out, we will

[jira] [Comment Edited] (SPARK-15815) Hang while enable blacklistExecutor and DynamicExecutorAllocator

2016-07-28 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15397271#comment-15397271 ] SuYan edited comment on SPARK-15815 at 7/28/16 8:50 AM: Current temp solution is

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

2016-07-06 Thread SuYan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15365543#comment-15365543 ] SuYan commented on SPARK-3630: -- may the reason was snappy 1.0.4.1 not support Concatenating? because the code

  1   2   >